A handy Database access library in Kotlin

Overview

KotliQuery

CI Builds Maven Central

KotliQuery is a handy RDB client library for Kotlin developers! The design is highly inspired by ScalikeJDBC, which is a proven database library in Scala. The priorities in this project are:

  • Less learning time
  • No breaking changes in releases
  • No additional complexity on top of JDBC

This library simply mitigates some pain points of the JDBC but our goal is not to completely encapsulate it.

Getting Started

The quickest way to try this library out would be to start with a simple Gradle project. You can find some examples here.

build.gradle

apply plugin: 'kotlin'

buildscript {
    ext.kotlin_version = '1.5.30'
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
    }
}
repositories {
    mavenCentral()
}
dependencies {
    implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
    implementation 'com.github.seratch:kotliquery:1.6.1'
    implementation 'com.h2database:h2:1.4.200'
}

Example

KotliQuery is much more easy-to-use than you expect. After just reading this short section, you will have learnt enough.

Creating DB Session

First thing you do is to create a Session object, which is a thin wrapper of java.sql.Connection instance. With this object, you can run queries using an established database connection.

import kotliquery.*

val session = sessionOf("jdbc:h2:mem:hello", "user", "pass") 

HikariCP

For production-grade applications, utilizing a connection pool library for better performance and resource management is highly recommended. KotliQuery provides an out-of-the-box solution that leverages HikariCP, which is a widely accepted connection pool library.

HikariCP.default("jdbc:h2:mem:hello", "user", "pass")

using(sessionOf(HikariCP.dataSource())) { session ->
   // working with the session
}

DDL Execution

You can use a session for executing both DDLs and DMLs. The asExecute method if a query object sets the underlying JDBC Statement method to execute.

session.run(queryOf("""
  create table members (
    id serial not null primary key,
    name varchar(64),
    created_at timestamp not null
  )
""").asExecute) // returns Boolean

Update Operations

Using asUpdate is an appropriate way to perform insert/update/delete statements. This method sets the underlying JDBC Statement method to executeUpdate.

val insertQuery: String = "insert into members (name,  created_at) values (?, ?)"

session.run(queryOf(insertQuery, "Alice", Date()).asUpdate) // returns effected row count
session.run(queryOf(insertQuery, "Bob", Date()).asUpdate)

Select Queries

Now that you've got a database table named members, it's time to run your first SQL statement with this library! To build a callable SQL executor, your code follows the three steps for it:

  • Use queryOf factory method with a query statement and its parameters to create a new Query object
  • Use #map method to attache a result extracting function ((Row) -> A) to the Query object
  • Specify the response type (asList/asSingle) for the result

The following query returns a list of all member's IDs. In this line, the SQL statement is not yet executed. Also, this object allIdsQuery does not have any state. This means that you can reuse th object multiple times.

val allIdsQuery = queryOf("select id from members").map { row -> row.int("id") }.asList

With a valid session object, you can perform the SQL statement. The type of returned ids would be safely determined by Kotlin compiler.

val ids: List<Int> = session.run(allIdsQuery)

As you see, the extractor function is greatly flexible. You can define functions with any return type. All you need to do is to implement a function that extracts values from JDBC ResultSet interator and map them into a single expected type value. Here is a complete example:

data class Member(
  val id: Int,
  val name: String?,
  val createdAt: java.time.ZonedDateTime)

val toMember: (Row) -> Member = { row -> 
  Member(
    row.int("id"), 
    row.stringOrNull("name"), 
    row.zonedDateTime("created_at")
  )
}

val allMembersQuery = queryOf("select id, name, created_at from members").map(toMember).asList
val members: List<Member> = session.run(allMembersQuery)

If you are sure that a query can return zero or one row, asSingle returns an optional single value as below:

val aliceQuery = queryOf("select id, name, created_at from members where name = ?", "Alice").map(toMember).asSingle
val alice: Member? = session.run(aliceQuery)

Technically, it's also possible to use asSingle along with an SQL statement returning multiple rows. With the default setting, the result data extraction returns only the first row in the results and skips the rest. In other words, KotliQuery silently ignores the inefficiency and the potential misbehavior. If you prefer detection by an error in this scenario, you can pass strict flag to Session initializer. With strict set to true, the query execution throws an exception if it detects multiple rows for asSingle.

// Session object constructor
val session = Session(HikariCP.dataSource(), strict = true)

// an auto-closing code block for session
using(sessionOf(HikariCP.dataSource(), strict = true)) { session ->

}

Named query parameters

An alternative way to bind parameters is to use named parameters that start with : in the statement string. Note that, with this feature, KotliQuery still uses a prepared statement internally and your query execution is safe from SQL injection. The parameter parts like :name and :age in the following example query won't be just replaced as string values.

queryOf(
  """
  select id, name, created_at 
  from members 
  where (name = :name) and (age = :age)
  """, 
  mapOf("name" to "Alice", "age" to 20)
)

Performance-wise, the named parameter syntax can be slightly slower for parsing the statement plus a tiny bit more memory-consuming. But for most use case, the overhead should be ignorable. If you would like to make your SQL statements more readable and/or if your query has to repeat the same parameter in a query, using named query parameters should improve your productivity and the maintainability of the query a lot.

Typed params

You can specify the Java type for each parameter in the following way. Passing the class Parameter helps KotliQuery properly determine the type to bind for each parameter in queries.

val param = Parameter(param, String::class.java)
queryOf("""select id, name 
    from members 
    where ? is null or ? = name""", 
    param, param)

As a handier way, you can use the following helper method.

queryOf("""select id, name 
    from members 
    where ? is null or ? = name""", 
    null.param<String>(), null.param<String>())

This functionality is particularly useful in the situations like the ones dsecribed here.

Working with Large Dataset

The #forEach allows you to work with each row with less memory consumption. With this way, your application code does not need to load all the query result data in memory at once. This feature is greatly useful when you load a large number of rows from a database table by a single query.

session.forEach(queryOf("select id from members")) { row ->
  // working with large data set
})

Transaction

Running queries in a transaction is of course supported! The Session object provides a way to start a transaction in a certain code block.

session.transaction { tx ->
  // begin
  tx.run(queryOf("insert into members (name,  created_at) values (?, ?)", "Alice", Date()).asUpdate)
}
// commit

session.transaction { tx ->
  // begin
  tx.run(queryOf("update members set name = ? where id = ?", "Chris", 1).asUpdate)
  throw RuntimeException() // rollback
}

As this library is a bit opinionated, transactions are available only with a code block. We intentionally do not support begin / commit methods. If you would like to manually manage the state of a transaction for some reason, you can use session.connection.commit() / session.connection.rollback() for it.

License

The MIT License Copyright (c) 2015 - Kazuhiro Sera

Comments
  • Bad support for `null` parameters in PortgreSQL

    Bad support for `null` parameters in PortgreSQL

    The issue has been described in 2011 here and I've just stumbled upon it in my project using kotliquery.

    There's a suggested way around it, using PreparedStatement.setObject(id, null, type) instead of PreparedStatement.setObject(id, null).

    opened by esaounkine 8
  • Make Session.transaction inline

    Make Session.transaction inline

    By marking it as an inline function, it becomes possible to use transaction from a coroutine context, without having to any additional sit-ups to make it work.

    For example, this code does not work now, but works fine with this patch applied:

    withTimeout(1000) {
        async {
            sessionOf(dataSource).use { dbSess ->
                dbSess.transaction { txSess ->
                    delay(1000)
                    // Without inline, fails with: 
                    //     Suspension functions can be called only within coroutine body
                    //
                    // Works fine when `transaction` is inline
                }
            }
        }
    }
    

    As far as I can tell, marking transaction as inline has no negative side effects for the current uses, and should be fully backwards compatible :)

    enhancement 
    opened by augustl 7
  • Why is count function not running ?

    Why is count function not running ?

    What do I want to do ?

    To count the rows returned by the query

    What works ?

    session.run(queryOf("select * from table-1").map { it.count() }.asSingle)
    

    Expected

    session.run(queryOf("select count(1) from table-1").map { it.int(0) }.asSingle)
    

    Am I doing anything wrong ?

    question 
    opened by raul1991 5
  • Modify named parameter extraction regex

    Modify named parameter extraction regex

    I am opening this PR as suggestion to fixing the issue that I reported in #36

    In order to prevent named param detection when using timestamp strings we are disallowing named parameters using a digit as first character in the name.

    I hope you find this solution appropriate as I really enjoy your lib and would like to continue using it. 🙏 If we can't fix this I have to revert to Spring JDBC to have native query support with named parameters.

    opened by joaomneto 5
  • issue 27, sql array

    issue 27, sql array

    Related to what we talked here https://github.com/seratch/kotliquery/issues/27

    I've added a createArrayOf method in the Session class. Kinda ugly, but at least we hide implementation details.

    I've also added an array method in Row, in order to retrieve a Kotlin array instead of a sql array. I'm not really sure if I did the right thing with the inline fun since I'm kinda new to Kotlin, but I didn't find other way to implement it. Let me know if there is a better way :)

    opened by rubendm92 4
  • Named parameters with array type

    Named parameters with array type

    Hi

    We are trying to do an insert to a table that contains a field of type text[] (postgres). We are doing something like this

    sessionProvider.createSession().use { session ->
        session.run(queryOf(
                "INSERT INTO t(x, xs) VALUES(:x, :xs)",
                mapOf("x" to "hello", "xs" to listOf("1", "2"))
        ).asUpdate)
    }
    

    We are getting an error because of the type os xs.

    org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of java.util.Collections$SingletonList. Use setObject() with an explicit Types value to specify the type to use.

    I don't see any example or documentation related to this. Is this supported?

    Thanks!

    opened by rubendm92 4
  • Timestamp fixes, uuid support

    Timestamp fixes, uuid support

    Closes #48 Plus:

    • Added tests
    • Added UUID support
    • Improved existed tests
    • Added ability to configure Hikari
    • Deprecated HikariCP.default(...) because redundant
    • Updated dependency versions
    • Applied same style for not imported classes
    bug 
    opened by ievgen-kapinos 3
  • session 'run' and 'execute' always returns false for 'create table' request

    session 'run' and 'execute' always returns false for 'create table' request

    You can check it for example here:

    https://github.com/seratch/kotliquery/blob/252016a5c25acbf569aa579c2daed116b1a9caac/sample/src/main/kotlin/Example.kt#L9-L15

    opened by Tolsi 3
  • Add shorthand sessionOf and transactionOf

    Add shorthand sessionOf and transactionOf

    Additional shorthand sessionOf that doesn't need to be wrapped with using.

    Plus transactionOf as often times you need a transaction for the whole session and with this you don't need to nest transaction block in session block.

    With this I believe it cannot get any better with the amount of control and readability it offers:

    val ds : DataSource by instance() // kodein
    
    sessionOf(ds) { session ->
      // ...
    }
    
    transactionOf(ds) { tx ->
      // ...
    }
    
    opened by Koriit 3
  • Support three types of batched jdbc statements.

    Support three types of batched jdbc statements.

    This is just a suggestion, I am fine with any changes you propose.

    JDBC batching is kind of awkward imo, and it does not fit very good with kotliquery (either). So after trying a bit, I decided that very specific api calls may be best, i.e. not relying on Query at all. I think this will fit quite well with what we need in our project.

    opened by stoyle 3
  • Support batched queries

    Support batched queries

    Hey. First off, been using kotliquery for almost a year, and it is working great for us. Like that it is just a relative small wrapper around jdbc, so thanks for your work.

    We have seen the need for batched queries, specifically batched inserts. Which I cannot see kotliquery supporting at the moment. Would that be a feature you would be interested in adding?

    We've implemented it ourselves, for our specific purpose, but I would be happy to contribute some code back.

    If not, could you make private inline fun <reified T> PreparedStatement.setTypedParam(idx: Int, param: Parameter<T>) public? This is the only part of the kotliquery code we had to copy to make it work for us.

    Cheers, Alf

    opened by stoyle 3
  • Own parsing of the batch insert/update result set

    Own parsing of the batch insert/update result set

    Another thought, the generated keys doesn't really have to be a long. They could be any type. I made it Long to be consistent with updateAndReturnGeneratedKey, but it would perhaps be more useful if it took an Row -> T as a param for mapping the generated keys from the resultset.

    Originally posted by @magott in https://github.com/seratch/kotliquery/issues/52#issuecomment-1082814620

    This (i.e - adding Row -> T) is actually very useful, instead of just receiving the ids As a matter of fact, it can save an additional roundtrip to the db, just to get the extra columns from that table that we batch-inserted values to.

    If this is something that can be done, I'd be happy to join as a reviewer.

    enhancement 
    opened by Somnium99 1
  • How to pass an indefinite amount of parameters to an sql query?

    How to pass an indefinite amount of parameters to an sql query?

    Hi there, Is there a right way to pass a list of parameters to an sql query? select * from table where id in ( 'a', 'list', 'of', 'ids' )

    Now I prepare a variable as String, but it looks unsecure.

    val query = """ select * from table where id in ( ${idList} ) """.trimMargin()
    l = kotliquery.queryOf(sqlMI).map(toMyObject).asList
    session.run(l)
    
    enhancement question 
    opened by cheprus 1
  • Usage of colon in a comment tries to bind it as a variable

    Usage of colon in a comment tries to bind it as a variable

    If I execute a query that has a comment that contains a colon, kotliquery attempts to use that colon as a bind parameter and we get the following error:

    The column index is out of range: 1, number of columns: 0.
    org.postgresql.util.PSQLException: The column index is out of range: 1, number of columns: 0.
    	at org.postgresql.jdbc.PgPreparedStatement.setNull(PgPreparedStatement.java:192)
    	at kotliquery.Session.populateParams$lambda-1(Session.kt:291)
    	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:721)
    	at kotliquery.Session.populateParams(Session.kt:91)
    	at kotliquery.Session.createPreparedStatement(Session.kt:116)
    	at kotliquery.Session.execute(Session.kt:204)
    	at kotliquery.action.ExecuteQueryAction.runWithSession(ExecuteQueryAction.kt:9)
    	at kotliquery.Session.run(Session.kt:231)
    

    Reproduction:

    val sql = """
    -- Some dumb comment :shrug:
    SELECT 1;
    """
    
    val url = "jdbc:postgresql://$host:$port/$database"
    val dbSession = sessionOf(url, username, password)
    
    using(dbSession) { session ->
        session.run(queryOf(sql, emptyMap()).asExecute)
    }
    

    Executing the same code straight against Postgres does not fail, as the commented out code is properly ignored.

    Kotliquery version: 1.6.1

    bug 
    opened by busches 3
  • Support for MySQL IN-Statement

    Support for MySQL IN-Statement

    Explaination

    In MySQL the IN operator allows you to determine if a value matches any value in a list of values. Unfortunately there is currently no support vor passing a list as a query parameter

    This works

    fun testSelect(): List<String> {
        val sql = """
            SELECT column_1, column_2 FROM TEST_TABLE WHERE column_1 IN ('a', 'b')
        """.trimIndent()
    
        val query = queryOf(sql).map { row -> row.string("column_2") }.asList
    
        return scanappDb.useConnection { dbSession ->
            dbSession.run(query)
        }
    }
    

    This doesn't work

    fun testSelect2(): List<String> {
        val sql = """
            SELECT column_1, column_2 FROM TEST_TABLE WHERE column_1 IN :values
        """.trimIndent()
    
        val query = queryOf(sql, mapOf("values" to arrayOf("a", "b")))
            .map { row -> row.string("column_2") }.asList
    
        return scanappDb.useConnection { dbSession ->
            dbSession.run(query)
        }
    }
    

    I tried:

    • Passing a list or array
    • Using Brackets "(" around :values
    • None of the combinations work

    Where it works

    For example room has an implementation for this: https://stackoverflow.com/questions/48406228/room-select-query-with-in-condition

    Final words

    It would be really nice if you could look into this issue :)

    enhancement 
    opened by MaaxGr 3
  • asUpdateAndReturnGeneratedKey does not work with postgresql returning id

    asUpdateAndReturnGeneratedKey does not work with postgresql returning id

    When we execute a postgresql insert statement that does returning id, we get the following exception: A result was returned when none was expected

    The below example will throw the exception: (Untested but created from other code that did)

     val insertQuery = "insert into test(name)  values (?) RETURNING id;"
    
    using(kotliquery.sessionOf(HikariCP.dataSource())){session ->
                    val query = queryOf(insertQuery,
                            "test name",
                    ).asUpdateAndReturnGeneratedKey
                    session.run(query)
            }
    

    Stacktrace:

    org.postgresql.util.PSQLException: A result was returned when none was expected.

    at org.postgresql.jdbc.PgStatement.getNoResultUpdateCount(PgStatement.java:254)
    at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:134)
    at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
    at kotliquery.Session$updateAndReturnGeneratedKey$1.invoke(Session.kt:158)
    at kotliquery.Session$updateAndReturnGeneratedKey$1.invoke(Session.kt:17)
    at kotliquery.LoanPattern.using(LoanPattern.kt:11)
    at kotliquery.PackageKt.using(package.kt:30)
    at kotliquery.Session.updateAndReturnGeneratedKey(Session.kt:157)
    at kotliquery.action.UpdateAndReturnGeneratedKeyQueryAction.runWithSession(UpdateAndReturnGeneratedKeyQueryAction.kt:9)
    at kotliquery.Session.run(Session.kt:175)
    
    bug question 
    opened by tbohnen 7
Owner
Kazuhiro Sera
Slack Bolt Framework / SDK Developer
Kazuhiro Sera
An Android library that makes developers use SQLite database extremely easy.

LitePal for Android 中文文档 LitePal is an open source Android library that allows developers to use SQLite database extremely easy. You can finish most o

Lin Guo 7.9k Dec 31, 2022
SquiDB is a SQLite database library for Android and iOS

Most ongoing development is currently taking place on the dev_4.0 branch. Click here to see the latest changes and try out the 4.0 beta. Introducing S

Yahoo 1.3k Dec 26, 2022
An Android library that makes developers use SQLite database extremely easy.

LitePal for Android 中文文档 LitePal is an open source Android library that allows developers to use SQLite database extremely easy. You can finish most o

Lin Guo 7.9k Jan 4, 2023
Collection of Kotlin APIs/tools to make using Realm Mobile database easier

Compass Kotlin API and tools to make working with Realm easier Components Compass is designed to make working with Realm easier through collection of

Arunkumar 16 Oct 4, 2022
Room Database Queries with Kotlin Flow

Room Database Queries with Flow This app displays a list of bus stops and arrival times. Tapping a bus stop on the first screen will display a list of

asifj96 0 Apr 26, 2022
Insanely easy way to work with Android Database.

Sugar ORM Insanely easy way to work with Android databases. Official documentation can be found here - Check some examples below. The example applicat

null 2.6k Dec 16, 2022
a 3d database ORM experiment. (used in two commercial projects)

Android-TriOrm a 3d database ORM experiment for Android. (used in two commercial projects). based around small tables concept and JVM Serialization. H

Tomer Shalev 19 Nov 24, 2021
Realm is a mobile database: a replacement for SQLite & ORMs

Realm is a mobile database that runs directly inside phones, tablets or wearables. This repository holds the source code for the Java version of Realm

Realm 11.4k Jan 2, 2023
Insanely easy way to work with Android Database.

Sugar ORM Insanely easy way to work with Android databases. Official documentation can be found here - Check some examples below. The example applicat

null 2.6k Jan 9, 2023
An Android helper class to manage database creation and version management using an application's raw asset files

THIS PROJECT IS NO LONGER MAINTAINED Android SQLiteAssetHelper An Android helper class to manage database creation and version management using an app

Jeff Gilfelt 2.2k Dec 23, 2022
Compile time processed, annotation driven, no reflection SQLite database layer for Android

SqliteMagic Simple yet powerful SQLite database layer for Android that makes database handling feel like magic. Overview: Simple, intuitive & typesafe

Siim Kinks 118 Dec 22, 2022
android 数据库框架,sqlite database

DBExecutor 主要的功能 1.使用了读写锁,支持多线程操作数据。 2.支持事务 3.支持ORM 4.缓存Sql,缓存表结构 这个类库主要用于android 数据库操作。 始终围绕着一个类对应一个表的概念。 只要创建一个实体类,就不用当心它怎么存储在数据库中,不用重新写增删改查的代码。基本的功

null 77 May 31, 2021
LiteOrm is a fast, small, powerful ORM framework for Android. LiteOrm makes you do CRUD operarions on SQLite database with a sigle line of code efficiently.

#LiteOrm:Android高性能数据库框架 A fast, small, powerful ORM framework for Android. LiteOrm makes you do CRUD operarions on SQLite database with a sigle line

马天宇 1.5k Nov 19, 2022
A simple ToDo app to demonstrate the use of Realm Database in android to perform some basic CRUD operations like Create, Update and Delete.

Creating a Realm Model Class @RealmClass open class Note() : RealmModel { @PrimaryKey var id: String = "" @Required var title: String

Joel Kanyi 15 Dec 18, 2022
An app with implementation of Room database for Android platform

Room Room An app with implementation of Room database for Android platform The Room persistence library provides an abstraction layer over SQLite to a

Void Hash 1 Jan 4, 2023
A Java/Kotlin library for Android platform, to manage bean's persistence in SQLite, SharedPreferences, JSON, XML, Properties, Yaml, CBOR.

Thanks to JetBrains for support Kripton Persistence Library project! Kripton Persistence Library Kripton is a java library, for Android platform, that

xcesco 117 Nov 11, 2022
Kotlin-Exposed-SQL - Example of using Exposed with Kotlin for the consumption of relational SQL Databases

Kotlin Exposed SQL Sencillo ejemplo sobre el uso y abuso de Exposed ORM de Jetbr

José Luis González Sánchez 3 Jun 14, 2022
Sprinkles is a boiler-plate-reduction-library for dealing with databases in android applications

Sprinkles Sprinkles is a boiler-plate-reduction-library for dealing with databases in android applications. Some would call it a kind of ORM but I don

Emil Sjölander 781 Nov 28, 2022
LiteGo is a Java-based asynchronous concurrency library. It has a smart executor, which can be freely set the maximum number of concurrent at same time , and the number of threads in waiting queue. It can also set waiting policies and overload strategies.

LiteGo:「迷你」的Android异步并发类库 LiteGo是一款基于Java语言的「异步并发类库」,它的核心是一枚「迷你」并发器,它可以自由地设置同一时段的最大「并发」数量,等待「排队」线程数量,还可以设置「排队策略」和「超载策略」。 LiteGo可以直接投入Runnable、Callable

马天宇 189 Nov 10, 2022