Simple, Expressive, Extensible Testing for Kotlin on the JVM

Duncan McGregor

Last update: Nov 24, 2022

Related tags

Overview

Minutest

JUnit multiplied by Kotlin

Why Another Test Framework?

JUnit is great for quickly writing and running tests as part of a TDD workflow, but try to do anything unusual and you have to reach for the documentation and specially written annotations.

Minutest extends JUnit with a simple model that allows you to solve your own problems with plain Kotlin.

For example

Conditionally running a test

JUnit has a special annotation

@Test
@EnabledIfEnvironmentVariable(named = "ENV", matches = "staging-server")
fun onlyOnStagingServer() {
    // ...
}

Minutest is just Kotlin

if (getenv("ENV") == "staging-server" ) test("only on staging server") {
    // ...
}

Parameterised tests

JUnit has three annotations

@DisplayName("Fruit tests")
@ParameterizedTest(name = "{index} ==> fruit=''{0}'', rank={1}")
@CsvSource("apple, 1", "banana, 2", "'lemon, lime', 3")
fun testWithCustomDisplayNames(fruit: String, rank, String) {
    // ...
}

Minutest is just Kotlin

context("Fruit tests") {
    listOf("apple" to 1, "banana" to 2, "lemon, lime" to 3).forEachIndexed { index, (fruit, rank) ->
        test("$index ==> fruit='$fruit', rank=$rank") {
            // ...
        }
    }
}

Nested Tests

JUnit needs more annotations

@DisplayName("A stack")
class TestingAStackDemo {

    var stack: Stack<Any> = Stack()

    @Nested
    @DisplayName("when new")
    inner class WhenNew {

        @Test
        fun `is empty`() {
            assertTrue(stack.isEmpty())
        }
    }

    @Nested
    @DisplayName("after pushing an element")
    inner class AfterPushing {

        var anElement = "an element"

        @BeforeEach
        fun pushAnElement() {
            stack.push(anElement)
        }

        @Test
        fun `it is no longer empty`() {
            assertFalse(stack.isEmpty())
        }
    }
}

../core/src/test/kotlin/dev/minutest/examples/StackExampleTestsJUnit.kt

Minutest is just Kotlin

class StackExampleTests : JUnit5Minutests {

    fun tests() = rootContext<Stack<Any>> {

        given { Stack() }

        context("when new") {

            test("is empty") {
                assertTrue(it.isEmpty())
            }
        }

        context("after pushing an element") {

            beforeEach {
                it.push("an element")
            }

            test("it is no longer empty") {
                assertFalse(it.isEmpty())
            }
        }
    }
}

../core/src/test/kotlin/dev/minutest/examples/StackExampleTests.kt

Minutest brings the power of Kotlin to JUnit, providing

A clean DSL to define nested contexts and tests
Generation and manipulation of tests at runtime
Much easier reuse of test code

For more information on how why Minutest is like it is, see My New Test Model .

Installation

Instructions

Moving from JUnit to Minutest

Here is a version of the JUnit 5 first test case, converted to Kotlin.

class MyFirstJUnitJupiterTests {

    private val calculator = Calculator()

    @Test
    fun addition() {
        calculator.add(2)
        assertEquals(2, calculator.currentValue)
    }

    @Test
    fun subtraction() {
        calculator.subtract(2)
        assertEquals(-2, calculator.currentValue)
    }
}

../core/src/test/kotlin/dev/minutest/examples/MyFirstJUnitJupiterTests.kt

In Minutest it looks like this

// Mix-in JUnit5Minutests to run Minutests with JUnit 5
//
// (JUnit 4 support is also available, see [JUnit4Minutests].)
class MyFirstMinutests : JUnit5Minutests {

    // tests are grouped in a context
    fun tests() = rootContext<Calculator> {

        // We need to tell Minutest how to build the fixture
        given { Calculator() }

        // define a test with a test block
        test("addition") {
            // inside tests, the fixture is `it`
            it.add(2)
            assertEquals(2, it.currentValue)
        }

        // each new test gets its own new fixture
        test("subtraction") { calculator ->
            subtract(2)
            assertEquals(-2, calculator.currentValue)
        }
    }
}

../core/src/test/kotlin/dev/minutest/examples/MyFirstMinutests.kt

Most tests require access to some state. The collection of state required by the tests is called the test fixture. In JUnit we use the fields of the test class as the fixture - in this case just the calculator. JUnit uses a fresh instance of the test class for each test method run, which is why the state of calculator after addition does not affect the result of subtraction.

Minutest does not create a fresh instance of the test class for each test, instead it invokes a fixture block in a context and passes the result into tests as this.

Tests for cooperating components will typically have more state than just the thing we are testing. In this case make the fixture hold all the state.

class ControlPanel(
    private val beep: () -> Unit,
    private val launchRocket: () -> Unit
) {
    private var keyTurned: Boolean = false

    fun turnKey() {
        keyTurned = true
    }

    fun pressButton() {
        if (keyTurned)
            launchRocket()
        else
            beep()
    }
    val warningLightOn get() = keyTurned
}

class CompoundFixtureExampleTests : JUnit5Minutests {

    // The fixture consists of all the state affected by tests
    class Fixture {
        var beeped = false
        var launched = false

        val controlPanel = ControlPanel(
            beep = { beeped = true },
            launchRocket = { launched = true }
        )
    }

    fun tests() = rootContext<Fixture> {
        given { Fixture() }

        context("key not turned") {
            test("light is off") {
                assertFalse(controlPanel.warningLightOn)
            }
            test("cannot launch when pressing button") {
                controlPanel.pressButton()
                assertTrue(beeped)
                assertFalse(launched)
            }
        }

        context("key turned") {
            beforeEach {
                controlPanel.turnKey()
            }
            test("light is on") {
                assertTrue(controlPanel.warningLightOn)
            }
            test("launches when pressing button") {
                controlPanel.pressButton()
                assertFalse(beeped)
                assertTrue(launched)
            }
        }
    }
}

../core/src/test/kotlin/dev/minutest/examples/fixtures/CompoundFixtureExampleTests.kt

Understanding fixtures is key to Minutest - read more

Run Code to Make Tests

The key to Minutest is that by separating the fixture from the test code, both are made available to manipulate as data.

For example, parameterised tests require special handling in JUnit, but not in Minutest.

class ParameterisedExampleTests : JUnit5Minutests {

    fun tests() = rootContext {

        context("palindromes") {

            // Creating a test for each of multiple parameters is as easy as
            // calling `test()` for each one.
            listOf("a", "oo", "racecar", "able was I ere I saw elba").forEach { candidate ->
                test("$candidate is a palindrome") {
                    assertTrue(candidate.isPalindrome())
                }
            }
        }
        context("not palindromes") {
            listOf("", "ab", "a man a plan a canal pananma").forEach { candidate ->
                test("$candidate is not a palindrome") {
                    assertFalse(candidate.isPalindrome())
                }
            }
        }

        // Minutest will check that the following tests are run
        willRun(
            "▾ tests",
            "  ▾ palindromes",
            "    ✓ a is a palindrome",
            "    ✓ oo is a palindrome",
            "    ✓ racecar is a palindrome",
            "    ✓ able was I ere I saw elba is a palindrome",
            "  ▾ not palindromes",
            "    ✓  is not a palindrome",
            "    ✓ ab is not a palindrome",
            "    ✓ a man a plan a canal pananma is not a palindrome"
        )
    }
}

fun String.isPalindrome(): Boolean =
    if (length == 0) false
    else (0 until length / 2).find { index -> this[index] != this[length - index - 1] } == null

../core/src/test/kotlin/dev/minutest/examples/ParameterisedExampleTests.kt

Reuse Tests

More complicated scenarios can be approached by writing your own function that returns a test or a context.

If you want to reuse the same tests for different concrete implementations, define a context with a function and call it for subclasses. Some people call this a contract.

// To run the same tests against different implementations, first define a ContextBuilder extension function
// that defines the tests you want run.
fun ContextBuilder<MutableCollection<String>>.behavesAsMutableCollection() {

    context("behaves as MutableCollection") {

        test("is empty when created") {
            assertTrue(isEmpty())
        }

        test("can add") {
            add("item")
            assertEquals("item", first())
        }
    }
}

// Now tests can supply the fixture and invoke the function to create the tests to verify the contract.
class ArrayListTests : JUnit5Minutests {

    fun tests() = rootContext<MutableCollection<String>> {
        given {
            ArrayList()
        }

        behavesAsMutableCollection()
    }
}

// We can reuse the contract for different collections.
class LinkedListTests : JUnit5Minutests {

    fun tests() = rootContext<MutableCollection<String>> {
        given {
            LinkedList()
        }

        behavesAsMutableCollection()
    }
}

../core/src/test/kotlin/dev/minutest/examples/ContractsExampleTests.kt

Structure Tests

When your tests grow so that they need more structure, Minutest has extensions to support Given When Then blocks

class ControlPanel(
    private val beep: () -> Unit,
    private val launchRocket: () -> Unit
) {
    private var keyTurned: Boolean = false

    fun turnKey() {
        keyTurned = true
    }

    fun pressButton(): Boolean =
        when {
            keyTurned -> {
                launchRocket()
                true
            }
            else -> {
                beep()
                false
            }
        }

    val warningLightOn get() = keyTurned
}

class ScenariosExampleTests : JUnit5Minutests {

    class Fixture {
        var beeped = false
        var launched = false

        val controlPanel = ControlPanel(
            beep = { beeped = true },
            launchRocket = { launched = true }
        )
    }

    fun tests() = rootContext<Fixture> {

        // Scenario defines a nested context
        Scenario("Cannot launch without key switch") {

            // GivenFixture sets up the fixture for the scenario
            GivenFixture("key not turned") {
                Fixture()
            }.Then("warning light is off") {
                // Then can check the setup
                assertFalse(controlPanel.warningLightOn)
            }

            // When performs the operation
            When("pressing the button") {
                controlPanel.pressButton()
            }.Then("result was false") { result ->
                // Then has access to the result
                assertFalse(result)
            }.And("it beeped") {
                // And makes another Thens with checks
                assertTrue(beeped)
            }.And("rocket was not launched") {
                assertFalse(launched)
            }
        }

        Scenario("Will launch with key switch") {
            GivenFixture("key turned") {
                Fixture().apply {
                    controlPanel.turnKey()
                }
            }.Then("warning light is on") {
                assertTrue(controlPanel.warningLightOn)
            }

            When("pressing the button") {
                controlPanel.pressButton()
            }.Then("result was true") { result ->
                assertTrue(result)
            }.And("it didn't beep") {
                assertFalse(beeped)
            }.And("rocket was launched") {
                assertTrue(launched)
            }
        }
    }
}

../core/src/test/kotlin/dev/minutest/examples/scenarios/ScenariosExampleTests.kt

Other Features

The Cookbook shows other ways to use Minutest.

Evolution

We're pretty happy with the core Minutest language and expect not to make any breaking changes without a major version update. Features like JUnit 4 support and test annotations are public but experimental - if you use anything in an experimental package you should expect it to change between minor releases, and move completely once adopted into the stable core.

Note that we aim for source and not binary compatibility. Some implementation may move from methods to extension functions, or from constructors to top level or companion-object functions.

Support

The best bet for feedback and help is the #minutest channel on the Kotlin Slack. See you there.

Comments

Split TestAnnotations into transforms and markers
This aims to solve the fundamental problems that I ran into because I was combining annotations with transforms.

I wanted to have TestAnnotation<in F> to allow TestAnnotation<Any?> (for example SKIP) to be combined with TestAnnotation<MyFixture>, but because a TestAnnotation carried a transform

val transform: NodeTransform<@UnsafeVariance F>? get()

I ended up with the ability to add a transform that could send the wrong fixture type in to a test - https://github.com/dmcg/minutest/blob/master/core/src/test/kotlin/dev/minutest/experimental/AnnotationTortureTests.kt#L76

What I've done is to separate transforms from markers, and have the ability to add either to a NodeBuilder https://github.com/dmcg/minutest/compare/annotations-split?expand=1#diff-158654f07c4f987b79c4f9694e9571e8L14

TestAnnotation now doesn't have any generic parameter, and can either add itself to the [NodeBuilder] as a marker, or a transform, or both.

interface TestAnnotation { fun applyTo(nodeBuilder: NodeBuilder<*>)

It can't though know anything about the fixture type. That doesn't stop it implementing SKIP - because SKIP ignores the fixture.

JUnit Rules support and fixture sequence flattening are simplified because they can directly add a transform to do their thing, rather than creating an annotation to add a transform. Note that inside a context block transforms have access to the fixture type, it's only TestAnnotations that don't.

So far so good?

There is one place that I can't make the typesystem work for this. TransformingAnnotation - https://github.com/dmcg/minutest/compare/annotations-split?expand=1#diff-a074010cc4b7550504b6d6291daefc88R15

I think that TransformingAnnotation is safe, because it takes a NodeTransform<*>, which can't assume anything about the fixture type. So I think that it should be safe to add that transform to a NodeBuilder<*> - but the compiler thinks different. I've shoehorned it in with a cast

class TransformingAnnotation( private val transform: NodeTransform<*> ) : TestAnnotation { override fun applyTo(nodeBuilder: NodeBuilder<*>) { nodeBuilder.addTransform(transform as (Node<out Any?>) -> Nothing) } }

Is this actually safe?

No, that isn't a rhetorical question, it's the reason that this is a PR rather than just smashed in to master.
opened by dmcg 6
root context with fixture arg

builds up on #30 to see the changes in this pull request, have a look at the second commit https://github.com/dmcg/minutest/commit/af7f4d658b8822069d5891a424daf1b75396ea01

opened by robstoll 3

Handling variance when reusing test cases

Example:

import dev.minutest.ContextBuilder
import dev.minutest.junit.JUnit5Minutests
import dev.minutest.rootContext
import org.junit.jupiter.api.Assertions.assertTrue

// contrived for example
class MySpecialList(delegate: List<Int>): List<Int> by delegate

private fun ContextBuilder<List<Int>>.respectsEmptyContract() {
  test("is empty") {
    assertTrue(isEmpty())
  }
  test("is empty (using fixture)") {
    assertTrue(fixture.isEmpty()) // fixture fails to compile for ContextBuilder<out List<Int>>
  }
}

internal class Mk : JUnit5Minutests {

  fun `testing list`() = rootContext<List<Int>> {
    fixture { emptyList() }
    respectsEmptyContract()
  }

  fun `testing my list`() = rootContext<MySpecialList> {
    fixture { MySpecialList(emptyList()) }
     // respectsEmptyContract() // fails to compile for `ContextBuilder<List<Int>>`

    derivedContext<List<Int>>("workaround casting") {
      deriveFixture { fixture }
      respectsEmptyContract()
    }
  }
}

The idea being able to use whatever tests provided by, in this example, respectsEmptyContract() for any List<Int> implementation (or really any List here, but not needed for the example).

// respectsEmptyContract() // fails to compile for `ContextBuilder<List<Int>>`

Fails to compile when ContextBuilder<List<Int>> is used.

  test("is empty (using fixture)") {
    assertTrue(fixture.isEmpty()) // fixture fails to compile for ContextBuilder<out List<Int>>
  }

Fails to compile (fixture fails) if ContextBuilder<out List<Int>> as the type bounds seem to go to Nothing

The workaround I have been using is to use a derivedContext with some casting:

private fun ContextBuilder<out List<Int>>.respectsEmptyContract() {
  derivedContext<List<Int>>("casting thing") {
    deriveFixture { this }
    test("is empty") {
      assertTrue(isEmpty())
    }
    test("is empty (using fixture)") {
      assertTrue(fixture.isEmpty()) // fixture fails to compile for ContextBuilder<out List<Int>>
    }
  }
}

Any suggestions?

opened by mkobit 3

Please re-add the @TestFactory feature again

In 0.26, I was able to create a TestClass with different methods. Some where annotated with @TestFactory, others with @Test.

It looks like the newer versions of minutest, no longer support this style of defining TestClasses. Is there any chance, to get this feature back?

Example:

internal class SomeRepoTest(
        @Autowired private val repo: SomeRepo
) : BootWebMockMvcTest() {
    companion object : KLogging(), SomeApiTestUtils

    @Test
    fun someTest() { ... }

   @TestFactory
    fun someTestFactory() = junitTests<Unit> { ... }

   @TestFactory
    fun anotherTestFactory() = junitTests<Unit> { ... }

}

opened by bastman 3

Test name printing on console running gradle is cryptic

Hey, thanks for the library.

When running gradle test the name of the tests is not the same as specified in the testcases. It is just tests()[number]

Please see the output:

io.gitlab.arturbosch.afk.uast.visit.TraversingSpec > allows to traverse back to UFile PASSED
io.gitlab.arturbosch.afk.uast.visit.UastRegistrarSpec > Given: registered UFile visit event PASSED
io.gitlab.arturbosch.afk.uast.visit.UastRegistrarSpec > When: two different lang UFiles are visited PASSED
io.gitlab.arturbosch.afk.uast.visit.UastRegistrarSpec > Then: the event counter must be two PASSED
io.gitlab.arturbosch.afk.uast.visit.UastVisitorSpec > prints names PASSED
io.gitlab.arturbosch.afk.uast.visit.TraversingTest > tests()[1] PASSED
io.gitlab.arturbosch.afk.uast.visit.TraversingTest > tests()[2] PASSED
io.gitlab.arturbosch.afk.uast.KotlinParsingSpec > tests()[1] PASSED
io.gitlab.arturbosch.afk.uast.KotlinParsingSpec > tests()[2] PASSED

The first four tests are written using spek2 and the last four minutest.

Gradle is 4.10.2, Kotlin 1.3, Junit 5.3.1, Junit-Platform 1.3.1, minutest 0.20.0, strikt 0.16.4

greetings.

opened by arturbosch 3

Please move to maven central. jcenter/bintray will be shutdown by 1st May
Hi,

it looks like jcenter/bintray will be shutdown by 1st May. Could you please consider publishing artifacts to maven central?

Thanks a lot :)

related article: https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

This might be of help as well. (Kluent moved from jcenter to maven central):

https://github.com/MarkusAmshove/Kluent/commit/15a8b454124ca49b9d1b89c1864417b7850c55ea
opened by bastman 2
Tests use classes instead of objects

JUnit 5 uses reflectomagic to instantiate test classes, which means that it instantiates multiple instances of an object's class, violating its singleton-ness. Therefore it's safer to define tests in classes.

opened by npryce 2
Reduce unchecked casts

builds up on #31 to see the changes in this pull request, have a look at the third commit https://github.com/dmcg/minutest/commit/91390236960a1dd55aadb4ea34608989aeb69fbd

opened by robstoll 1
An abstract sealed class hieararchy modelling runtime contexts and tests

Allows navigation of the context/test hierarchy to be public while the implementation details of how fixtures are instantiated is internal to the library.

No need for star projections when navigating the test hierarchy.

(as discussed on Slack)

opened by npryce 1

Experimental SKIP feature unnecessary context evaluation

Version: 1.7.0

I was trying out the dev.minutest.experimental.SKIP feature and noticed that it looks like deriveFixture/modifyFixture/probably other pieces are executed for a nested context that is skipped. I was using Mockito mocks in a fixture and was modifying the behavior in each nested context. At the most nested context, I had a handful of tests that I wanted to skip because they were not implemented yet, so I thought I would try out SKIP. I saw test failures due to using strict mocks, so the expectations set on the mocks were throwing org.mockito.exceptions.misusing.UnnecessaryStubbingException after the test because it looks like the behavior settings were called even though the nested context was skipped.

My expectation is that the whole buildup would not be executed for skipped contexts.

Here is an example that I think demonstrates my expectations better than my explanation:

import dev.minutest.experimental.SKIP
import dev.minutest.experimental.minus
import dev.minutest.junit.JUnit5Minutests
import dev.minutest.rootContext

internal class Mk : JUnit5Minutests {
    fun tests() = rootContext<Int> {
        fixture { 0 }
        context("l1") {
            deriveFixture {
                println("deriveFixture: ${it.name}")
                this + 1
            }
            context("l2") {
                test("l2 test") {
                    println(it.name)
                }
            }

            SKIP - context("l2 skipped") {
                test("l2 skipped test") {
                    println(it.name)
                }
            }
        }
    }
}

Output:

deriveFixture: l2 test
l2 test
deriveFixture: l2 skipped

Test ignored.

dev.minutest.experimental.MinutestSkippedException
	at dev.minutest.experimental.InexcludingKt$skipped$1.invoke(inexcluding.kt:47)
	at dev.minutest.experimental.InexcludingKt$skipped$1.invoke(inexcluding.kt)
	at dev.minutest.Test.invoke(Node.kt)
	at dev.minutest.Test.invoke(Node.kt:37)
	at dev.minutest.internal.PreparedContext$runTest$1.invoke(PreparedContext.kt:21)
	at dev.minutest.internal.PreparedContextKt.tryMap(PreparedContext.kt:61)
	at dev.minutest.internal.PreparedContextKt.access$tryMap(PreparedContext.kt:1)
	at dev.minutest.internal.PreparedContext.runTest(PreparedContext.kt:21)
	at dev.minutest.internal.TestExecutor$andThen$1$runTest$testletForParent$1.invoke(TestExecutor.kt:25)
	at dev.minutest.internal.TestExecutor$andThen$1$runTest$testletForParent$1.invoke(TestExecutor.kt:17)
	at dev.minutest.internal.PreparedContext$runTest$1.invoke(PreparedContext.kt:21)
	at dev.minutest.internal.PreparedContextKt.tryMap(PreparedContext.kt:61)
	at dev.minutest.internal.PreparedContextKt.access$tryMap(PreparedContext.kt:1)
	at dev.minutest.internal.PreparedContext.runTest(PreparedContext.kt:21)
	at dev.minutest.internal.TestExecutor$andThen$1$runTest$testletForParent$1.invoke(TestExecutor.kt:25)
	at dev.minutest.internal.TestExecutor$andThen$1$runTest$testletForParent$1.invoke(TestExecutor.kt:17)
	at dev.minutest.internal.RootExecutor.runTest(TestExecutor.kt:36)
	at dev.minutest.internal.TestExecutor$andThen$1.runTest(TestExecutor.kt:28)
	at dev.minutest.internal.TestExecutor$andThen$1.runTest(TestExecutor.kt:28)
	at dev.minutest.internal.TestExecutor$DefaultImpls.runTest(TestExecutor.kt:12)
	at dev.minutest.internal.TestExecutor$andThen$1.runTest(TestExecutor.kt:17)
	at dev.minutest.junit.JUnit5MinutestsKt$toDynamicNode$1.execute(JUnit5Minutests.kt:59)

...

My expectation is that

deriveFixture: l2 skipped

would not need to be called since those tests are skipped.

opened by mkobit 2

Disallow duplicate names within the same context

Either tests with the same name, subcontexts with the same name, or a test with the same name as a subcontext.

    @org.junit.jupiter.api.Test
    fun `names are unique within a context`() {
        assertFailsWith<IllegalArgumentException>("tests") {
            rootContext<Unit> {
                test("nameclash") {}
                test("nameclash") {}
            }.buildNode()
        }
        assertFailsWith<IllegalArgumentException>("contexts") {
            rootContext<Unit> {
                context("nameclash") {}
                context("nameclash") {}
            }.buildNode()
        }
        assertFailsWith<IllegalArgumentException>("context and test") {
            rootContext<Unit> {
                context("nameclash") {}
                test("nameclash") {}
            }.buildNode()
        }
    }

opened by npryce 0