Kotlin multiplatform benchmarking toolkit

Related tags

kotlin benchmark gradle jvm jmh
Overview

JetBrains incubator project GitHub license Build status Maven Central Gradle Plugin Portal

NOTE:   Starting from version 0.3.0 of the library:

  • The library runtime is published to Maven Central and no longer published to Bintray.
  • The Gradle plugin is published to Gradle Plugin Portal
  • The Gradle plugin id has changed to org.jetbrains.kotlinx.benchmark
  • The library runtime artifact id has changed to kotlinx-benchmark-runtime

NOTE:   When Kotlin 1.5.0 or newer is used make sure the kotlin-gradle-plugin is pulled from Maven Central, not Gradle Plugin Portal. For more information: https://github.com/Kotlin/kotlinx-benchmark/issues/42

kotlinx.benchmark is a toolkit for running benchmarks for multiplatform code written in Kotlin and running on the next supported targets: JVM, JavaScript.

Technically it can be run on Native target, but current implementation doesn't allow to get right measurements in many cases for native benchmarks, so it isn't recommended to use this library for native benchmarks yet. See issue for more information.

If you're familiar with JMH, it is very similar and uses it under the hoods to run benchmarks on JVM.

Requirements

Gradle 6.8 or newer

Kotlin 1.4.30 or newer

Gradle plugin

Use plugin in build.gradle:

plugins {
    id 'org.jetbrains.kotlinx.benchmark' version '0.3.1'
}

For Kotlin/JS specify building nodejs flavour:

kotlin {
    js {
        nodejs()
        …
    }   
}

For Kotlin/JVM code, add allopen plugin to make JMH happy. Alternatively, make all benchmark classes and methods open.

For example, if you annotated each of your benchmark classes with @State(Scope.Benchmark):

@State(Scope.Benchmark)
class Benchmark {
    …
}

and added the following code to your build.gradle:

plugins {
    id 'org.jetbrains.kotlin.plugin.allopen'
}

allOpen {
    annotation("org.openjdk.jmh.annotations.State")
}

then you don't have to make benchmark classes and methods open.

Runtime Library

You need a runtime library with annotations and code that will run benchmarks.

Enable Maven Central for dependencies lookup:

repositories {
    mavenCentral()
}

Add the runtime to dependencies of the platform source set, e.g.:

kotlin {
    sourceSets {
        commonMain {
             dependencies {
                 implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.3.1")
             }
        }
    }
}

Configuration

In a build.gradle file create benchmark section, and inside it add a targets section. In this section register all targets you want to run benchmarks from. Example for multiplatform project:

benchmark {
    targets {
        register("jvm") 
        register("js")
        register("native")
    }
}

This package can also be used for Java and Kotlin/JVM projects. Register a Java sourceSet as a target:

benchmark {
    targets {
        register("main") 
    }
}

To configure benchmarks and create multiple profiles, create a configurations section in the benchmark block, and place options inside. Toolkit creates main configuration by default, and you can create as many additional configurations, as you need.

benchmark {
    configurations {
        main { 
            // configure default configuration
        }
        smoke { 
            // create and configure "smoke" configuration, e.g. with several fast benchmarks to quickly check
            // if code changes result in something very wrong, or very right. 
        }       
    }
}

Available configuration options:

  • iterations – number of measuring iterations
  • warmups – number of warm up iterations
  • iterationTime – time to run each iteration (measuring and warmup)
  • iterationTimeUnit – time unit for iterationTime (default is seconds)
  • outputTimeUnit – time unit for results output
  • mode – "thrpt" for measuring operations per time, or "avgt" for measuring time per operation
  • include("…") – regular expression to include benchmarks with fully qualified names matching it, as a substring
  • exclude("…") – regular expression to exclude benchmarks with fully qualified names matching it, as a substring
  • param("name", "value1", "value2") – specify a parameter for a public mutable property name annotated with @Param
  • reportFormat – format of report, can be json(default), csv, scsv or text

Time units can be NANOSECONDS, MICROSECONDS, MILLISECONDS, SECONDS, MINUTES, or their short variants such as "ms" or "ns".

Example:

benchmark {
    // Create configurations
    configurations {
        main { // main configuration is created automatically, but you can change its defaults
            warmups = 20 // number of warmup iterations
            iterations = 10 // number of iterations
            iterationTime = 3 // time in seconds per iteration
        }
        smoke {
            warmups = 5 // number of warmup iterations
            iterations = 3 // number of iterations
            iterationTime = 500 // time in seconds per iteration
            iterationTimeUnit = "ms" // time unity for iterationTime, default is seconds
        }   
    }
    
    // Setup targets
    targets {
        // This one matches compilation base name, e.g. 'jvm', 'jvmTest', etc
        register("jvm") {
            jmhVersion = "1.21" // available only for JVM compilations & Java source sets
        }
        register("js") {
            // Note, that benchmarks.js uses a different approach of minTime & maxTime and run benchmarks
            // until results are stable. We estimate minTime as iterationTime and maxTime as iterationTime*iterations
        }
        register("native")
    }
}

Separate source sets for benchmarks

Often you want to have benchmarks in the same project, but separated from main code, much like tests. Here is how:

Define source set:

sourceSets {
    benchmarks
}

Propagate dependencies and output from main sourceSet.

dependencies {
    benchmarksCompile sourceSets.main.output + sourceSets.main.runtimeClasspath 
}

You can also add output and compileClasspath from sourceSets.test in the same way if you want to reuse some of the test infrastructure.

Register benchmarks source set:

benchmark {
    targets {
        register("benchmarks")    
    }
}

Examples

The project contains examples subproject that demonstrates using the library.

Issues
  • Multi-format reports

    Multi-format reports

    • CSV/SCSV/TEXT JMH-similar formatter for JS/Native
    • possibility to select format from gradle plugin
    • fixes #34
    • printing summary after suite execution on all platforms
    opened by whyoleg 13
  • Trouble with Kotlin/Native

    Trouble with Kotlin/Native

    I have a simple project in Idea for Kotlin/Native and next build.gradle file:

    plugins {
        id "kotlinx.benchmark" version "0.2.0-dev-8"
        id 'org.jetbrains.kotlin.multiplatform' version '1.3.61'
        id 'java'
        id 'org.jetbrains.kotlin.plugin.allopen' version "1.3.61"
    }
    repositories {
        mavenCentral()
        maven { url 'https://dl.bintray.com/kotlin/kotlinx' }
    }
    dependencies {
        implementation "org.jetbrains.kotlinx:kotlinx.benchmark.runtime:0.2.0-dev-8"
    }
    kotlin {
        // For ARM, should be changed to iosArm32 or iosArm64
        // For Linux, should be changed to e.g. linuxX64
        // For MacOS, should be changed to e.g. macosX64
        // For Windows, should be changed to e.g. mingwX64
        mingwX64("mingw") {
            binaries {
                executable {
                    // Change to specify fully qualified name of your application's entry point:
                   entryPoint = 'sample.main'
                    // Specify command-line arguments, if necessary:
                    runTask?.args('')
                }
            }
        }
        sourceSets {
            // Note: To enable common source sets please comment out 'kotlin.import.noCommonSourceSets' property
            // in gradle.properties file and re-import your project in IDE.
            mingwMain {
            }
            mingwTest {
            }
        }
    }
    
    // Use the following Gradle tasks to run your application:
    // :runReleaseExecutableMingw - without debug symbols
    // :runDebugExecutableMingw - with debug symbols
    
    benchmark {
        configurations {
            main {
                iterations = 10
                warmups = 1
            }
        }
        targets {
            register("mingw")
        }
    }
    
    
    allOpen {
        annotation("org.openjdk.jmh.annotations.State")
    }
    

    When I run benchmark task in Gradle, I got:

    Testing started at 21:18 ...
    > Configure project :
    Kotlin Multiplatform Projects are an experimental feature.
    Warning: Cannot find a benchmark compilation 'native', ignoring.
    > Task :compileKotlinMingw UP-TO-DATE
    > Task :mingwProcessResources NO-SOURCE
    > Task :mingwMainKlibrary UP-TO-DATE
    > Task :mingwBenchmarkGenerate FAILED
    FAILURE: Build failed with an exception.
    * What went wrong:
    Execution failed for task ':mingwBenchmarkGenerate'.
    > There was a failure while executing work items
       > A failure occurred while executing kotlinx.benchmark.gradle.NativeSourceGeneratorWorker
          > e: Failed to resolve Kotlin library: stdlib
    * Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
    * Get more help at https://help.gradle.org
    BUILD FAILED in 4s
    

    What I am doing wrong?

    Wha I am trying:

    1. Remove allopen
    2. Add additional stdlib
    3. Remve stdlib from External Libraries in Idea (it returns)
    opened by vldF 6
  • Declare system properties reads to be compatible with Gradle configuration cache

    Declare system properties reads to be compatible with Gradle configuration cache

    Gradle reports direct system properties reads as configuration cache problems as system properties are potential build configuration inputs. See https://docs.gradle.org/7.0/userguide/configuration_cache.html for more details

    opened by ALikhachev 4
  • Could not find or load main class kotlinx.benchmark.jvm.JvmBenchmarkRunnerKt

    Could not find or load main class kotlinx.benchmark.jvm.JvmBenchmarkRunnerKt

    I am trying to use kotlinx-benchmark in an open source project. I followed all the instructions but I am getting the following error when I try to run the benchmarks from the Gradle panel in IntelliJ IDEA:

    Could not find or load main class kotlinx.benchmark.jvm.JvmBenchmarkRunnerKt
    

    My changes to build.gradle.kts can be seen in the following pull-request:

    https://github.com/tree-ware/tree-ware-kotlin-core/pull/13/files

    I am interested only in JVM benchmarks. I tried adding a dependency to org.jetbrains.kotlinx:kotlinx-benchmark-runtime-jvm but that did not help either.

    opened by deepak-nulu 3
  • Gradle 6 compatibility

    Gradle 6 compatibility

    Currently using with gradle 6 gives:

    Execution failed for task ':examples:benchmarksBenchmarkGenerate'.
    > There was a failure while executing work items
       > A failure occurred while executing kotlinx.benchmark.gradle.JmhBytecodeGeneratorWorker
          > Generation of JMH bytecode failed with 1errors:
              - Group name should be the legal Java identifier.
               [[email protected]]
    
    opened by altavir 3
  • Missing incubator badge in the Jetbrains confluence

    Missing incubator badge in the Jetbrains confluence

    You have added a JB incubator badge to the readme which links to https://confluence.jetbrains.com/display/ALL/JetBrains+on+GitHub

    But the incubator explanation can't be found there... Is the link wrong? Need it to be added to the confluence?

    opened by StefMa 2
  • Publish to maven central instead of bintray

    Publish to maven central instead of bintray

    Bintray is being sunsetted: https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

    opened by jdai8 2
  • Wrong configuration for 0.2.0-dev-2

    Wrong configuration for 0.2.0-dev-2

    Dependency configuration requires 0.2.0-dev-2, not 0.2.0. Also it requires maven ("https://dl.bintray.com/orangy/maven") because of dependency on non-published kotlinx-cli.

    opened by altavir 2
  • Implement Blackhole

    Implement Blackhole

    opened by qurbonzoda 2
  • Can't build when used with Kotlin experimental features

    Can't build when used with Kotlin experimental features

    The library I am benchmarking uses Kotlin's ExperimentalUnsignedTypes. I have already added

    @OptIn(ExperimentalUnsignedTypes::class)
    

    to the benchmark class and

    tasks.withType<KotlinCompile>().configureEach {
        kotlinOptions.freeCompilerArgs += "-Xopt-in=kotlin.RequiresOptIn"
    }
    

    to build.gradle.kts but still got the following Gradle error:

    Execution failed for task ':app:mainBenchmarkGenerate'.
    > There was a failure while executing work items
       > A failure occurred while executing kotlinx.benchmark.gradle.JmhBytecodeGeneratorWorker
          > Generation of JMH bytecode failed with 1errors:
              - Group name should be the legal Java identifier.
               [[email protected]]
    
    * Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
    
    
    
    opened by ShreckYe 0
  • `@Setup(Level.Invocation)` on Native?

    `@Setup(Level.Invocation)` on Native?

    Support @Setup(Level.Invocation) on Native.

    Currently, @Setup has no parameters.

    enhancement 
    opened by CommanderTvis 1
  • Explore the possibility of making the plugin an ordinary module

    Explore the possibility of making the plugin an ordinary module

    On current moment, plugin is not in classpath of integration module, because plugin is included build (to use current version of plugin in examples). We can add it to classpath in some way, but now kotlinx.benchmark.gradle.BenchmarkConfiguration depends on BenchmarksExtension, looks like we need to remove this dependency to reuse it.

    I have a question, why is included build is used at all? Maybe it's possible to make it as an ordinary module, and after that integration module will be moved just to integrationTest source set under it (the same way as it's done in kotlinx-atomicfu and binary-compatibility-validator gradle plugins)

    Originally posted by @whyoleg in https://github.com/Kotlin/kotlinx-benchmark/pull/37#discussion_r623687850

    opened by qurbonzoda 0
  • Version 0.3 not working with kotlin 1.5.* MPP plugin

    Version 0.3 not working with kotlin 1.5.* MPP plugin

    The gradle file I use (rewritten from examples/kotlin-mpp removing the infra stuff):

    plugins {
        id 'org.jetbrains.kotlin.multiplatform' version "1.5.10"
        id 'org.jetbrains.kotlin.plugin.allopen' version "1.5.0"
        id 'org.jetbrains.kotlinx.benchmark' version "0.3.1"
    }
    
    // how to apply plugin to a specific source set?
    allOpen {
        annotation("org.openjdk.jmh.annotations.State")
    }
    
    repositories {
        mavenCentral()
    }
    
    kotlin {
        jvm {
            compilations.all {
                kotlinOptions {
                    jvmTarget = '1.8'
                }
            }
        }
    
        js {
            nodejs {
                
            }
        }
    
        sourceSets.all {
            languageSettings {
                progressiveMode = true
                useExperimentalAnnotation("kotlin.Experimental")
                useExperimentalAnnotation('kotlin.ExperimentalUnsignedTypes')
            }
        }
    
        sourceSets {
            commonMain {
                dependencies {
                    //classpath group: 'org.antlr', name: 'antlr4-runtime', version: '4.9.2'
                    implementation "org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.3.1"
                }
            }
            commonTest {
                dependencies {
                    implementation 'org.jetbrains.kotlin:kotlin-test-common'
                    implementation 'org.jetbrains.kotlin:kotlin-test-annotations-common'
                }
            }
            jvmMain {}
            jvmTest {
                dependencies {
                    implementation 'org.jetbrains.kotlin:kotlin-test'
                    implementation 'org.jetbrains.kotlin:kotlin-test-junit'
                }
            }
            jsMain {}
            jsTest {
                dependencies {
                    implementation 'org.jetbrains.kotlin:kotlin-test-js'
                }
            }
    
            nativeMain {
                dependsOn commonMain
                dependencies {
                }
            }
        }
    }
    
    // Configure benchmark
    benchmark {
        configurations {
            main { // --> jvmBenchmark/jsBenchmark + benchmark
                iterations = 5 // number of iterations
                iterationTime = 300
                iterationTimeUnit = "ms"
                advanced("forks", 3)
            }
    
            params {
                iterations = 5 // number of iterations
                iterationTime = 300
                iterationTimeUnit = "ms"
                include("ParamBenchmark")
                param("data", 5, 1, 8)
                param("unused", 6, 9)
            }
    
            fast { // --> jvmFastBenchmark
                include("Common")
                exclude("long")
                iterations = 1
                iterationTime = 300 // time in ms per iteration
                iterationTimeUnit = "ms" // time in ms per iteration
                advanced("forks", 1)
            }
    
            csv {
                include("Common")
                exclude("long")
                iterations = 1
                iterationTime = 300
                iterationTimeUnit = "ms"
                reportFormat = "csv" // csv report format
            }
        }
    
        // Setup configurations
        targets {
            // This one matches compilation base name, e.g. 'jvm', 'jvmTest', etc
            register("jvm") {
                jmhVersion = "1.21"
            }
            register("js")
            register("native")
            register("macosX64")
            register("linuxX64")
            register("mingwX64")
        }
    }
    

    This does not run, gradle benchmark resulting in a very cryptic error:

    FAILURE: Build failed with an exception.
    * What went wrong:
    Execution failed for task ':jsBenchmarkPackageJson'.
    > org/antlr/v4/runtime/Lexer
    

    This is, in fact, a j.l.ClassNotFoundException exception in plugin. Adding the following buildscript header fixes the issue:

    buildscript {
        repositories {
            mavenCentral()
        }
    
        dependencies {
            classpath group: 'org.antlr', name: 'antlr4-runtime', version: '4.9.2'
        }
    }
    

    So this is, indeed, a missing dependency in plugin or something similar

    opened by belyaev-mikhail 1
  • Exclude duplicate classes for Jar task

    Exclude duplicate classes for Jar task

    Hi,

    This pull request attempts to fix issue #39, which causes a failure for benchmarks with complex dependencies when using Gradle 7. This issue is caused by the default duplicatesStrategy of the Jar task triggering a failure due to complex dependencies (where the same class might exist multiple times).

    The solution is to update the duplicatesStrategy for the jmhBenchmarkJar tasks to exclude duplicate classes.

    Fixes #39

    opened by fabianishere 0
  • Wrong treatment of dublication META-INF elements

    Wrong treatment of dublication META-INF elements

    The jvmBenchmarkJar task is failing with complex dependencies on Gradle 7 unless the following fix is applied:

        val jvmBenchmarkJar by tasks.getting(org.gradle.jvm.tasks.Jar::class) {
            duplicatesStrategy = org.gradle.api.file.DuplicatesStrategy.EXCLUDE
        }
    
    opened by altavir 1
  • Support JS IR in plugin

    Support JS IR in plugin

    • fixes #30
    • suppresses #31
    • TODO ms time units fails on IR
    opened by whyoleg 8
  • Kotlin/Native support

    Kotlin/Native support

    opened by LepilkinaElena 0
  • Turn on JS IR

    Turn on JS IR

    I'm having such an error because the runtime library doesn't support JS IR.

    Caused by: org.gradle.internal.component.NoMatchingConfigurationSelectionException: No matching variant of org.jetbrains.kotlinx:kotlinx.benchmark.runtime:0.2.0-dev-20 was found. The consumer was configured to find a usage of 'kotlin-runtime' of a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'js', attribute 'org.jetbrains.kotlin.js.compiler' with value 'ir' but:
      - Variant 'js-api' capability org.jetbrains.kotlinx:kotlinx.benchmark.runtime:0.2.0-dev-20 declares a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'js':
    
    opened by CommanderTvis 0
  • js target requires benchmark class to be annotated with State annotation

    js target requires benchmark class to be annotated with State annotation

    Good description of what jmh state is for.

    It doesn't seem correct that for the js benchmark to run you must annotate the class with @State(Scope.Benchmark).

    I believe the only requirement should be for the benchmark functions to be annotated with @Benchmark.

    If it is an intentional requirement, then I think it should be mentioned in the README.

    opened by Munzey 0
Releases(v0.3.1)
  • v0.3.1(Apr 30, 2021)

  • v0.3.0(Mar 1, 2021)

    • Require the minimum Kotlin version of 1.4.30
    • Require the minimum Gradle version of 6.8
    • Change runtime artifact id from kotlinx.benchmark.runtime to kotlinx-benchmark-runtime
    • Publish runtime to Maven Central instead of Bintray #33
    • Change plugin id from kotlinx.benchmark to org.jetbrains.kotlinx.benchmark
    • Change plugin artifact id from kotlinx.benchmark.gradle to kotlinx-benchmark-plugin
    • Publish plugin to Gradle Plugin Portal instead of Bintray
    Source code(tar.gz)
    Source code(zip)
Owner
Kotlin
Kotlin Tools and Libraries
Kotlin
Dependency Injection library for Kotlin Multiplatform, support iOS and Android

Multiplatform-DI library for Kotlin Multiplatform Lightweight dependency injection framework for Kotlin Multiplatform application Dependency injection

Anna Zharkova 12 Jul 27, 2021
Kotlin Multiplatform Mobile App Template

KMMT : Kotlin Multiplatform Mobile Template Kotlin Multiplatform Mobile Development Simplified KMMT is a KMM based project template designed to simpli

Jitty Andiyan 117 Jul 19, 2021
BuildConfig for Kotlin Multiplatform Project

BuildKonfig BuildConfig for Kotlin Multiplatform Project. It currently supports embedding values from gradle file. Table Of Contents Motivation Usage

Yasuhiro SHIMIZU 187 Jul 28, 2021
Kotlin multiplatform benchmarking toolkit

NOTE: Starting from version 0.3.0 of the library: The library runtime is published to Maven Central and no longer published to Bintray. The Gradle plu

Kotlin 167 Jul 28, 2021
DI can be simple. Forget about modules and components. Just use it!

PopKorn - Kotlin Multiplatform DI PopKorn is a simple, powerful and lightweight Kotlin Multiplatform Dependency Injector. It doesn't need any modules

Pau Corbella 109 May 22, 2021
Generic AST parsing library for kotlin multiplatform

kotlinx.ast kotlinx.ast is a generic AST (Abstract Syntax Tree) parsing library, Kotlin is currently the only supported language. The library is desig

null 140 Jul 27, 2021
KaMP Kit by Touchlab is a collection of code and tools designed to get your mobile team started quickly with Kotlin Multiplatform.

KaMP Kit Welcome to the KaMP Kit! About Goal The goal of the KaMP Kit is to facilitate your evaluation of Kotlin Multiplatform (aka KMP). It is a coll

Touchlab 1.2k Aug 2, 2021
A collaborative list of awesome jetpack compose resources.

Awesome Jetpack compose A collaborative list of awesome jetpack compose resources. Featured in Content Official Documentation Conference talks Article

Naveen T P 280 Jul 17, 2021
Kotlin Multiplatform project that gets network data from Food2Fork.ca

Food2Fork Recipe App This is the codebase for a Kotlin Multiplatform Mobile course. [Watch the course](https://codingwithmitch.com/courses/kotlin-mult

Mitch Tabian 181 Jul 24, 2021
A Bluetooth kotlin multiplatform "Cross-Platform" library for iOS and Android

Blue-Falcon A Bluetooth "Cross Platform" Kotlin Multiplatform library for iOS, Android, MacOS, Raspberry Pi and Javascript. Bluetooth in general has t

Andrew Reed 146 Jul 24, 2021
A collection of hand-crafted extensions for your Kotlin projects.

Splitties Splitties is a collection of small Kotlin multiplatform libraries (with Android as first target). These libraries are intended to reduce the

Louis CAD 1.8k Aug 4, 2021
The most essential libraries for Kotlin Multiplatform development

Essenty The most essential libraries for Kotlin Multiplatform development. Supported targets: android jvm js (IR and LEGACY) iosArm64, iosX64 watchosA

Arkadii Ivanov 13 Aug 2, 2021
Design Patterns implemented in Kotlin

Design Patterns In Kotlin ⚠️ New article about testing: Unit Testing with Mockito 2 Project maintained by @dbacinski (Dariusz Baciński) Inspired by De

Dariusz Baciński 4.8k Jul 28, 2021
Minecraft NBT support for kotlinx.serialization

knbt An implementation of Minecraft's NBT format for kotlinx.serialization. Technical information about NBT can be found here. Using the same version

Ben Woodworth 11 Jul 1, 2021