A sample repo describing best practices for snapshot testing on Android

Overview

Road to effective snapshot testing

A sample repo describing best practices for snapshot testing on Android. This includes for now:

  1. Parameterized Tests: Write the test once and run it under all desired configurations. This includes snapshot tests under various Font sizes
  2. Filtered Parameterized Tests: Create filters to group the configurations, so that we can run only the tests corresponding to a certain group. As described later in this ReadMe, this can be useful when having a large bunch of snapshot tests, in order to reduce building times. That way we can select to run a group of them on every PR (e.g. the essential ones), and the rest once a day.

The code here will reflect everything discussed on my series on snapshot testing

All the samples use Shot from Karumi, what is a great library that facilitates snasphot testing. I've been using an emulator with API 23 for a matter of simplicity. That is because Shot requires less configuration for API < 29.

Parameterized Snapshot Tests

They enable us to test a given view with all possible different configurations, namely:

  1. Theme (Light and Dark)
  2. Font Size
  3. Languages
  4. View dimensions (width and height)
  5. Specific view states
  6. Others...

With Parameterized snapshot test we can write the test once and run it for all the desired configurations! This can be achieved by using org.junit.runners.Parameterized as in the example below.

  1. Create a TestItem which encapsulates all the info we want to inject as parameter. I strongly recommend to add a testName to it, which must be unique among all tests. This is used to distinguish each snapshot file from the rest.
class TestItem(
    val fontScale: FontScale,
    val theme: DialogTheme,
    val texts: Array<Int>,
    val testName: String
)
  1. Write the method that inflates the view and takes/compares the screenshot. Use the parameters from TestItem to set the view into the right state.
private fun ScreenshotTest.snapDeleteDialog(testItem: TestItem) {
    val activity = launch(MainActivity::class.java)
        .waitForActivity(testItem.theme.themId)

    val dialog = waitForView {
        DialogBuilder.buildDeleteDialog(
            activity,
            onDeleteClicked = {/* no-op*/ },
            bulletTexts = itemArray(testItem.texts)
        )
    }

    compareScreenshot(dialog, name = testItem.testName, widthInPx = 800)
}

If you also want to enable different Font size configurations to your tests, you can add the FontSizeTestRule library to your project. We will see in the next section how to use it.

  1. Finally, we need to add the parameters. We will use org.junit.runners.Parameterized for that. We will pass the TestItem in the class constructor, and define each single configuration to be run inside a method annotated with @Parameterized.Parameters.
@RunWith(Parameterized::class)
class AllDeleteDialogSnapshotTest(private val testItem: TestItem) : ScreenshotTest {

    @get:Rule
    val fontSize = FontScaleRules.fontScaleTestRule(testItem.fontScale)

    companion object {
        @JvmStatic
        @Parameterized.Parameters
        fun data(): Array<TestItem> =
            arrayOf(
                TestItem(
                    SMALL,
                    MATERIAL_DARK,
                    arrayOf(R.string.shortest),
                    "DARK_SMALL"
                ),
                TestItem(
                    HUGE,
                    MATERIAL_DARK,
                    repeatedItem(7, R.string.largest),
                    "DARK_HUGE"
                )
            )
    }

    @Test
    fun snapDialog() {
        snapDeleteDialog(testItem)
    }
}

Now you can just run your test with ./gradlew executeScreenshotTests -Precord and you should see the results!

Filter Parameterized Tests

Running snapshot tests for all configurations on every PR can be very time-consuming and lead to incrementally slower builds. One approach to solve that issue is to run only a part of those tests on every PR (e.g. the most common config, our "smoke" tests) and all snapshot tests - or "all non-smoke tests" - once a day (e.g. during the night or whenever it disturbs your team the least). In doing so, we get informed of the most important visual regression bugs on every PR (i.e. blocking bugs), and still get notified of the non-blocking bugs once a day.

We can accomplish this by filtering our tests with -Pandroid.testInstrumentationRunnerArguments.annotation=com.your.package.YourAnnotation or -Pandroid.testInstrumentationRunnerArguments.notAnnotation=com.your.package.YourAnnotation

Warning: These arguments are supported by org.junit.runners.Parameterized, but not by all runners, e.g. JUnitParams fails if used.

So first of all, we need to create our annotation

annotation class SmokeTest

And then we add another Test Class that reuses our previously defined TestItem and snapDeleteDialog(testItem) containing the "most common config" parameters, like here below

@RuWith(Parameterized::class)
class BasicDeleteDialogSnapshotTest(private val testItem: TestItem) : ScreenshotTest {

    @get:Rule
    val fontSize = FontScaleRules.fontScaleTestRule(testItem.fontScale)

    companion object {
        @JvmStatic
        @Parameterized.Parameters
        fun data(): Array<TestItem> = arrayOf(
            TestItem(
                NORMAL,
                MATERIAL_LIGHT,
                arrayOf(R.string.largest, R.string.middle, R.string.shortest),
                "SMOKE"
            )
        )
    }

    @SmokeTest
    @Test
    fun snapDialog() {
        snapDeleteDialog(testItem)
    }
}

Take a look at DelegateDialogTest.kt to see how these Parameterized Tests are implemented and run ./gradlew executeScreenshotTests -Precord -Pandroid.testInstrumentationRunnerArguments.annotation=com.example.road.to.effective.snapshot.testing.utils.SmokeTest to verify that only the @SmokeTest runs!

What is coming next:

  1. More advance samples
  2. Tips to remove flakiness
  3. Tips to increase test execution speed
  4. Jetpack compose samples and more...
Comments
  • Repo test reorganization and rename + Dropshots & Paparazzi tests

    Repo test reorganization and rename + Dropshots & Paparazzi tests

    • Rename "Road to effective snapshot testing" to "Android screenshot testing playground" containing screenshot tests with the libraries:
      1. Dropshots
      2. Paparazzi
      3. Shot
    • move screen related UI components/dialogs into their own UI library module (required by Paparazzi)
    • create submodules in each UI module for each screenshot testing library (:dropshots, :paparazzi, :shot)
    • Write the equivalent (as much as possible) existing Shot tests also with Paparazzi and Dropshots
    opened by sergio-sastre 0
  • AndroidUITestingUtils addition + TestParameterInjector

    AndroidUITestingUtils addition + TestParameterInjector

    1. Add examples for parameterized snapshot tests with AndroidUiTestingUtils for:
    • Locale
    • Font sizes
    • Orientation
    • UI mode
    1. Add parameterized snapshot tests with TestParameterInjector
    2. Add example for screenshot testing Activities
    opened by sergio-sastre 0
  • Code clean up

    Code clean up

    The code was adapted from a personal project and had therefore a lot of commented code and leftovers.

    This task should clean up and structure the code a bit

    opened by sergio-sastre 0
  • Disable animations automatically: no need for doing it manually on the emulator settings

    Disable animations automatically: no need for doing it manually on the emulator settings

    Specially when running on the CI, it is cumbersome to do that manually. We can use TestButler for that, and add also an option not to use testButler -> animations won't be disabled automatically

    Add also documentation on how to do it

    opened by sergio-sastre 0
  • Update to AGP 7.x.x and do corresponding adjustment for Shot compatibility + update Shot for windows support

    Update to AGP 7.x.x and do corresponding adjustment for Shot compatibility + update Shot for windows support

    Shot requires some adjustments to run on AGP 7.x.x (configure applicationId) Additionally, shot supports windows since 5.12.2 and FontSizeTestRule has a new version

    opened by sergio-sastre 0
  • Modularise app for better extensibility

    Modularise app for better extensibility

    Right now everything is in one module. ideally we should create one module for each action e.g. Every button in the main screen opens an activity -> module

    1. Module for snapshot testing a Dialog
    2. Module for snapshot testing a RecyclerView ...
    opened by sergio-sastre 0
Owner
Sergio Sastre Flórez
I make things happen
Sergio Sastre Flórez
Snapshot/Screenshot test example project

Snapshot Snapshot/Screenshot test example code using Showkase (https://github.com/airbnb/Showkase) Paparazzi (https://github.com/cashapp/paparazzi) Te

Anders Ullnæss 3 Nov 25, 2022
Powerful, elegant and flexible test framework for Kotlin with additional assertions, property testing and data driven testing

Kotest is a flexible and comprehensive testing tool for Kotlin with multiplatform support. To learn more about Kotest, visit kotest.io or see our quic

Kotest 3.8k Jan 3, 2023
Selenium locators for Java/Kotlin that resemble the Testing Library (testing-library.com).

Selenium Testing Library Testing Library selectors available as Selenium locators for Kotlin/Java. Why? When I use Selenium, I don't want to depend on

Luís Soares 5 Dec 15, 2022
Android UI Testing

User scenario testing for Android Robotium is an Android test automation framework that has full support for native and hybrid applications. Robotium

null 2.8k Dec 14, 2022
A set of AssertJ helpers geared toward testing Android.

AssertJ Android A set of AssertJ assertions geared toward testing Android. Deprecated The support libraries and play services are developing at a rate

Square 1.6k Jan 3, 2023
Android Unit Testing Framework

Robolectric is the industry-standard unit testing framework for Android. With Robolectric, your tests run in a simulated Android environment inside a

Robolectric 5.6k Jan 3, 2023
Android UI Testing

User scenario testing for Android Robotium is an Android test automation framework that has full support for native and hybrid applications. Robotium

null 2.7k Apr 8, 2021
Testify — Android Screenshot Testing

Testify — Android Screenshot Testing Add screenshots to your Android tests Expand your test coverage by including the View-layer. Testify allows you t

ndtp 43 Dec 24, 2022
A programmer-oriented testing framework for Java.

JUnit 4 JUnit is a simple framework to write repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks. For more infor

JUnit 8.4k Jan 9, 2023
A programmer-oriented testing framework for Java.

JUnit 4 JUnit is a simple framework to write repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks. For more infor

JUnit 8.4k Dec 28, 2022
Turbine is a small testing library for kotlinx.coroutines Flow.

A small testing library for kotlinx.coroutines Flow

Cash App 1.8k Jan 5, 2023
Fixtures for Kotlin providing generated values for unit testing

A tool to generate well-defined, but essentially random, input following the idea of constrained non-determinism.

Appmattus Limited 191 Dec 21, 2022
Morsa: Jetpack Compose UI Testing Framework

Morsa: Jetpack Compose UI Testing Framework Test library to ease UI testing with Jetpack Compose Purpose This library aims to add some useful wrappers

HyperDevs 10 Dec 3, 2022
Lovely Systems Database Testing

Lovely DB Testing This repository provides opinionated testing helpers for database testing used at Lovely Systems. License This plugin is made availa

Lovely Systems GmbH 1 Feb 23, 2022
Project for testing intern candidate simple level

RickAndMorty-TestTask Тестовый проект 'Гайд по мультфильму Рик и Морти' для практикантов начального и продвинутого уровня. Структура проекта Структура

null 0 Nov 18, 2021
Very simple Morse API text translator. Mainly to do some testing with jitpack, API development etc.

Very simple Morse text translator API. Far from finish or even being reliable. Use for single sentence. Mainly to test github dependency for Android

Piotr Dymala 0 Dec 30, 2021
A set of TestRules and ActivityScenarios to facilitate UI Testing under given configurations: FontSizes, Locales

AndroidUiTestingUtils A set of TestRules, ActivityScenarios and utils to facilit

Sergio Sastre Flórez 122 Dec 23, 2022
Android Sample Kotlin+ MVVM + Coroutines + Retrofit +Hilt+ Junit + Mockito

NTimes AppSample NY Times Most Popular Articles simple app to hit the NY Times Most Popular Articles API and show a list of articles, that shows detai

Amer Elsayed 0 Dec 27, 2021
This is a sample API project for Rest Assured with Maven Kotlin DSL & JUnit 5

Kotlin-RestAssured Test Automation This is a sample API project for Rest Assured with Maven Kotlin DSL & JUnit 5 Introduction All the Test Cases kept

Dilshan Fernando 0 Dec 9, 2021