Easily scale your Android Instrumentation Tests across Firebase Test Lab with Flank.

Related tags

Testing android testing flank
Overview
Comments
  • Fladle plugin to automatically configure additional test apks

    Fladle plugin to automatically configure additional test apks

    It would be nice if fladle automatically configured and ran all your project's additional test apks in a single fladle run in order to aggregate the test reporting.

    Gathering thoughts here on creating a secondary fladle plugin that would be applied in the project root and detect all the modules with androidTest apks. These test apks would then be passed to additionalTestApks as arguments.

    Considerations:

    • APKs without tests should not be passed to flank since it will cause flank to fail.
    • Does ordering matter when passing to additionalTestApks?
    • How should multiple application apks be ordered with respect to other modules?

    If you are currently using `additionalTestApks, could you post a snippet as to how yours in your project looks below? I want to get an idea of how the community is currently using that feature so that this plugin can mimic that behavior.

    enhancement 
    opened by runningcode 22
  • Configuration option to ignore failures

    Configuration option to ignore failures

    Standard test tasks support ignoreFailures, which Flank doesn't appear to. Ignoring failures is useful for CI, where it's more desirable for Gradle invocations to complete successfully, and test results to be analyzed afterward.

    opened by damianw 21
  • Yaml formatting error with projectId

    Yaml formatting error with projectId

    Hi, we came across a very weird error while using fladle on a new project. Our project id on Firebase has dash - in it. And this creates an issue parsing the configuration since dash is a reserved char. According to my google search, this actually depends on yaml parser.

    I was thinking of creating a PR but I was not sure what is the best way to handle this. Should we just wrap the value into quotes?

    # instead of this
  projectId: some-funny-name

# this
  projectId: "some-funny-name"

    Here is the error I receive:

    > Task :app:execFlankCriticalPath
version: v21.03.1
revision: **redacted**
session id: **redacted**

Invalid value for [project]: no argument value found
    opened by tasomaniac 20
  • Fulladle doesn't work without a service account

    Fulladle doesn't work without a service account

    I am trying Fladle for the first time, and ran into an issue when setting up Fulladle for multi-module testing. If the service account is not specified it throws an error

    > Task :writeConfigProps FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':writeConfigProps'.
> ServiceAccountCredentials in fladle extension not set. https://github.com/runningcode/fladle#serviceaccountcredentials

    Can we please have the ability to run Fulladle as well without the service account?

    opened by uberbinge 20
  • Feature Request: Generate app apk and test apk if not found

    Feature Request: Generate app apk and test apk if not found

    Right now I have to build app and test apks before running flank:

    ./gradlew clean app:assembleDebug app:assembleDebugAndroidTest app:runFlank --no-parallel

    I know that you already added "auto detect apks" in #1. It would be great if Fladle was one step smarter and I could just call ./gradlew app:runFlank and have it work out all the magic.

    opened by tir38 15
  • Update current flank flags

    Update current flank flags

    New flags added

    • runTimeout
    • ignoreFailedTests
    • disableSharding
    • smartFlankDisableUpload
    • testRunnerClass
    • localResultsDir
    • numUniformShards
    • clientDetails
    • testTargetsAlwaysRun
    • otherFiles
    • networkProfile
    opened by pawelpasterz 13
  • Support for running multiple runFlank tasks with configs

    Support for running multiple runFlank tasks with configs

    I noticed that if I try to run multiple runFlank tasks for different configs, only the config for the last runFlank task is used

    For example, if I have a build.gradle file that does this:

    ...

fladle {
    smokeTest {
        testTargets = ['annotation SmokeTest']
    }
    expandedSmokeTest {
        testTargets = ['annotation ExpandedSmokeTest']
    }
}

    and then try to run

    ./gradlew :app:runFlankSmokeTests :app:runFlankExpandedSmokeTests

    The output would be something like:

    > Task :app:execFlankSmokeTest
version: v20.06.1
revision: ...
AndroidArgs
    gcloud:
      ...
      test-targets:
        - annotation ExpandedSmokeTest
...
> Task :app:execFlankExpandedSmokeTest
version: v20.06.1
revision: ...
AndroidArgs
    gcloud:
      ...
      test-targets:
        - annotation ExpandedSmokeTest

    Digging into the gradle log, I think this might be caused by the yaml file being overwritten because I see these lines sequentially:

    > Task :app:writeConfigPropsSmokeTest
> Task :app:writeConfigPropsExpandedSmokeTest
    opened by SamThompson 12
  • dependOnAssemble = true builds all variants

    dependOnAssemble = true builds all variants

    dependOnAssemble = true builds all variants of an application rather than just the one flank needs to run the tests.

    This appears to be related to #60, if fladle doesn't know what variant apks it needs, it makes sense that it wouldn't know which to build.

    opened by AndrewReitz 11
  • Support for Robo Tests without Robo script

    Support for Robo Tests without Robo script

    This is an enhancement request to support robo tests without a robo script. Firebase Test Lab supports this, although it appears fladle does not. In the fladle source, the decision to do a robo test is based on the roboscript file path being non-null.

    Issues sometimes come up with R8, ProGuard, or DexGuard that make robo testing the my release APK helpful in catching issues that might occur during first launch. However, DexGuard also obfuscates resource IDs which makes creating a roboscript for a release build much more difficult.

    Being able to launch a robo test without a roboscript will enable my use case, but would also lower the barrier for others who want to run a robo test without recording a script.

    opened by ccjernigan 10
  • Publish plugin marker artifact to Sonatype

    Publish plugin marker artifact to Sonatype

    Version 0.9.2-SNAPSHOT cannot be applied using plugins DSL because marker artifact is missing from Sonatype.

    // settings.gradle:
pluginManagement {
    repositories {
        maven {
            url "https://oss.sonatype.org/content/repositories/snapshots/"
            mavenContent {
                snapshotsOnly()
            }
        }
        gradlePluginPortal()
    }
    plugins {
        id "com.osacky.fladle" version "0.9.2-SNAPSHOT"
    }
}

// build.gradle:
plugins {
    id "com.osacky.fladle" 
}

    Plugin [id: 'com.osacky.fladle', version: '0.9.2-SNAPSHOT'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Included Builds (None of the included builds contain this plugin)
- Plugin Repositories (could not resolve plugin artifact 'com.osacky.fladle:com.osacky.fladle.gradle.plugin:0.9.2-SNAPSHOT')
  Searched in the following repositories:
    maven(https://oss.sonatype.org/content/repositories/snapshots/)
    Gradle Central Plugin Repository
    opened by technoir42 9
  • Test matrices pass, but flank fails

    Test matrices pass, but flank fails

    Hi, we're seeing some flaky behavior from flank/fladle in our CI. Sometimes, all matrices are reported to have passed, but the flank process exits with a non-zero status code and tanks the build. Rebuilding the project succeeds, usually.

    Any idea what we could do to debug this? It's not clear from the output what, if anything, is actually wrong.

    Gradle output 

    > Task :app:execFlank
Picked up JAVA_TOOL_OPTIONS: -Xmx1g
AndroidArgs
    gcloud:
      results-bucket: test-lab-ysvv7d00td10a-hppdbavjfw5v2
      results-dir: null
      record-video: true
      timeout: 20m
      async: false
      results-history-name: null
      # Android gcloud
      app: /home/circleci/project/app/build/outputs/apk/debug/app-debug.apk
      test: /home/circleci/project/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
      auto-google-login: false
      use-orchestrator: true
      directories-to-pull:

      performance-metrics: true
      test-runner-class: null
      test-targets:

      device:
        - model: Pixel2
          version: 28
          locale: en_US
          orientation: portrait
      flaky-test-attempts: 0

    flank:
      max-test-shards: 30
      shard-time: -1
      repeat-tests: 1
      smart-flank-gcs-path: 
      smart-flank-disable-upload: false
      files-to-download:

      test-targets-always-run:

      disable-sharding: false
      project: app-ae416
      local-result-dir: results
      # Android Flank Yml
      additional-app-test-apks:

RunTests
  Uploading app-debug-androidTest.apk .  Uploading app-debug.apk .


  Smart Flank cache hit: 0% (0 / 389)
  Shard times: 120s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s, 130s

  389 tests / 30 shards

  30 matrix ids created in 0m 7s
  <bucket url>

PollMatrices
  0m  0s matrix-f88t4kmt0fhaa Pixel2-28 VALIDATING
  0m 45s matrix-f88t4kmt0fhaa Pixel2-28 PENDING
  1m 31s matrix-f88t4kmt0fhaa Pixel2-28 Starting attempt 1.
  1m 31s matrix-f88t4kmt0fhaa Pixel2-28 RUNNING
  2m 32s matrix-f88t4kmt0fhaa Pixel2-28 Started logcat recording.
  2m 32s matrix-f88t4kmt0fhaa Pixel2-28 Preparing device.
  3m  2s matrix-f88t4kmt0fhaa Pixel2-28 Installing apps.
  3m 18s matrix-f88t4kmt0fhaa Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  3m 18s matrix-f88t4kmt0fhaa Pixel2-28 Retrieving Performance Environment information from the device.
  3m 18s matrix-f88t4kmt0fhaa Pixel2-28 Started crash detection.
  3m 18s matrix-f88t4kmt0fhaa Pixel2-28 Started crash monitoring.
  3m 18s matrix-f88t4kmt0fhaa Pixel2-28 Started performance monitoring.
  3m 18s matrix-f88t4kmt0fhaa Pixel2-28 Started video recording.
  3m 18s matrix-f88t4kmt0fhaa Pixel2-28 Starting instrumentation test.
  5m 33s matrix-f88t4kmt0fhaa Pixel2-28 Completed instrumentation test.
  5m 33s matrix-f88t4kmt0fhaa Pixel2-28 Stopped performance monitoring.
  6m  4s matrix-f88t4kmt0fhaa Pixel2-28 Stopped crash monitoring.
  6m  4s matrix-f88t4kmt0fhaa Pixel2-28 Stopped logcat recording.
  6m  4s matrix-f88t4kmt0fhaa Pixel2-28 Retrieving Post-test Package Stats information from the device.
  6m  4s matrix-f88t4kmt0fhaa Pixel2-28 Done. Test time = 132 (secs)
  6m  4s matrix-f88t4kmt0fhaa Pixel2-28 Starting results processing. Attempt: 1
  6m 20s matrix-f88t4kmt0fhaa Pixel2-28 Completed results processing. Time taken = 11 (secs)
  6m 20s matrix-f88t4kmt0fhaa Pixel2-28 FINISHED
  6m 20s matrix-f88t4kmt0fhaa FINISHED
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Starting attempt 1.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Started logcat recording.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Preparing device.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Installing apps.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Retrieving Performance Environment information from the device.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Started crash detection.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Started crash monitoring.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Started performance monitoring.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Started video recording.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Starting instrumentation test.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Completed instrumentation test.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 Stopped performance monitoring.
  6m 20s matrix-bl70fygtj6zla Pixel2-28 RUNNING
  6m 50s matrix-bl70fygtj6zla Pixel2-28 Stopped crash monitoring.
  6m 50s matrix-bl70fygtj6zla Pixel2-28 Retrieving Post-test Package Stats information from the device.
  6m 50s matrix-bl70fygtj6zla Pixel2-28 Stopped logcat recording.
  6m 50s matrix-bl70fygtj6zla Pixel2-28 Done. Test time = 166 (secs)
  6m 50s matrix-bl70fygtj6zla Pixel2-28 Starting results processing. Attempt: 1
  6m 50s matrix-bl70fygtj6zla Pixel2-28 Completed results processing. Time taken = 11 (secs)
  6m 50s matrix-bl70fygtj6zla Pixel2-28 FINISHED
  6m 50s matrix-bl70fygtj6zla PENDING
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Starting attempt 1.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Started logcat recording.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Preparing device.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Installing apps.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Retrieving Performance Environment information from the device.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Started crash detection.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Started crash monitoring.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Started performance monitoring.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Started video recording.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Starting instrumentation test.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Completed instrumentation test.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 Stopped performance monitoring.
  6m 50s matrix-37kmr3c6f8e32 Pixel2-28 RUNNING
  7m  5s matrix-37kmr3c6f8e32 Pixel2-28 Stopped crash monitoring.
  7m  5s matrix-37kmr3c6f8e32 Pixel2-28 Stopped logcat recording.
  7m  5s matrix-37kmr3c6f8e32 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m  5s matrix-37kmr3c6f8e32 Pixel2-28 Done. Test time = 200 (secs)
  7m  5s matrix-37kmr3c6f8e32 Pixel2-28 Starting results processing. Attempt: 1
  7m 21s matrix-37kmr3c6f8e32 Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 21s matrix-37kmr3c6f8e32 Pixel2-28 FINISHED
  7m 21s matrix-37kmr3c6f8e32 FINISHED
  7m 21s matrix-8dxthyvybquna Pixel2-28 Starting attempt 1.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Started logcat recording.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Preparing device.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Installing apps.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Retrieving Performance Environment information from the device.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Started crash detection.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Started performance monitoring.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Started crash monitoring.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Started video recording.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Starting instrumentation test.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Completed instrumentation test.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Stopped performance monitoring.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Stopped crash monitoring.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Stopped logcat recording.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 21s matrix-8dxthyvybquna Pixel2-28 Done. Test time = 175 (secs)
  7m 21s matrix-8dxthyvybquna Pixel2-28 Starting results processing. Attempt: 1
  7m 21s matrix-8dxthyvybquna Pixel2-28 Completed results processing. Time taken = 10 (secs)
  7m 21s matrix-8dxthyvybquna Pixel2-28 FINISHED
  7m 21s matrix-8dxthyvybquna FINISHED
  7m 22s matrix-3jglg60p1485w Pixel2-28 Starting attempt 1.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Started logcat recording.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Preparing device.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Installing apps.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Retrieving Performance Environment information from the device.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Started crash detection.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Started crash monitoring.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Started performance monitoring.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Started video recording.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Starting instrumentation test.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Completed instrumentation test.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Stopped performance monitoring.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Stopped crash monitoring.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Stopped logcat recording.
  7m 22s matrix-3jglg60p1485w Pixel2-28 Done. Test time = 180 (secs)
  7m 22s matrix-3jglg60p1485w Pixel2-28 Starting results processing. Attempt: 1
  7m 22s matrix-3jglg60p1485w Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 22s matrix-3jglg60p1485w Pixel2-28 FINISHED
  7m 22s matrix-3jglg60p1485w FINISHED
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Starting attempt 1.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Started logcat recording.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Preparing device.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Installing apps.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Retrieving Performance Environment information from the device.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Started crash detection.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Started performance monitoring.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Started crash monitoring.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Started video recording.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Starting instrumentation test.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Completed instrumentation test.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Stopped performance monitoring.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Stopped crash monitoring.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Stopped logcat recording.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Done. Test time = 166 (secs)
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Starting results processing. Attempt: 1
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 22s matrix-1jjpnoxqjp62l Pixel2-28 FINISHED
  7m 22s matrix-1jjpnoxqjp62l FINISHED
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Starting attempt 1.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Started logcat recording.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Preparing device.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Installing apps.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Retrieving Performance Environment information from the device.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Started crash detection.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Started crash monitoring.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Started performance monitoring.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Started video recording.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Starting instrumentation test.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Completed instrumentation test.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Stopped performance monitoring.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Stopped crash monitoring.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Stopped logcat recording.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Done. Test time = 152 (secs)
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Starting results processing. Attempt: 1
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 23s matrix-22g4mj42dxnrd Pixel2-28 FINISHED
  7m 23s matrix-22g4mj42dxnrd FINISHED
  7m 23s matrix-ercommuadp4aa Pixel2-28 Starting attempt 1.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Started logcat recording.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Preparing device.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Installing apps.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Retrieving Performance Environment information from the device.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Started crash detection.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Started performance monitoring.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Started crash monitoring.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Started video recording.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Starting instrumentation test.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Completed instrumentation test.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Stopped performance monitoring.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Stopped crash monitoring.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Stopped logcat recording.
  7m 23s matrix-ercommuadp4aa Pixel2-28 Done. Test time = 131 (secs)
  7m 23s matrix-ercommuadp4aa Pixel2-28 Starting results processing. Attempt: 1
  7m 23s matrix-ercommuadp4aa Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 23s matrix-ercommuadp4aa Pixel2-28 FINISHED
  7m 23s matrix-ercommuadp4aa FINISHED
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Starting attempt 1.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Started logcat recording.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Preparing device.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Installing apps.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Retrieving Performance Environment information from the device.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Started crash detection.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Started performance monitoring.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Started crash monitoring.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Started video recording.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Starting instrumentation test.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Completed instrumentation test.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Stopped performance monitoring.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Stopped crash monitoring.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Stopped logcat recording.
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Done. Test time = 152 (secs)
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Starting results processing. Attempt: 1
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 Completed results processing. Time taken = 10 (secs)
  7m 23s matrix-161ng2wpsmxyk Pixel2-28 FINISHED
  7m 23s matrix-161ng2wpsmxyk FINISHED
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Starting attempt 1.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Started logcat recording.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Preparing device.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Installing apps.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Retrieving Performance Environment information from the device.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Started crash detection.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Started performance monitoring.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Started crash monitoring.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Started video recording.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Starting instrumentation test.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Completed instrumentation test.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Stopped performance monitoring.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Stopped crash monitoring.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Stopped logcat recording.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Done. Test time = 132 (secs)
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Starting results processing. Attempt: 1
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 24s matrix-2hdlxiv9c6lrj Pixel2-28 FINISHED
  7m 24s matrix-2hdlxiv9c6lrj FINISHED
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Starting attempt 1.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Started logcat recording.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Preparing device.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Installing apps.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Started crash detection.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Started crash monitoring.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Started performance monitoring.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Started video recording.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Starting instrumentation test.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Completed instrumentation test.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Stopped performance monitoring.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Stopped crash monitoring.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Stopped logcat recording.
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Done. Test time = 161 (secs)
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Starting results processing. Attempt: 1
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 Completed results processing. Time taken = 10 (secs)
  7m 24s matrix-1jzvaawt57tg4 Pixel2-28 FINISHED
  7m 24s matrix-1jzvaawt57tg4 FINISHED
  7m 25s matrix-60veg8z79b5va Pixel2-28 Starting attempt 1.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Started logcat recording.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Preparing device.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Installing apps.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Retrieving Performance Environment information from the device.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Started crash detection.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Started performance monitoring.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Started crash monitoring.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Started video recording.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Starting instrumentation test.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Completed instrumentation test.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Stopped performance monitoring.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Stopped crash monitoring.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Stopped logcat recording.
  7m 25s matrix-60veg8z79b5va Pixel2-28 Done. Test time = 163 (secs)
  7m 25s matrix-60veg8z79b5va Pixel2-28 Starting results processing. Attempt: 1
  7m 25s matrix-60veg8z79b5va Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 25s matrix-60veg8z79b5va Pixel2-28 FINISHED
  7m 25s matrix-60veg8z79b5va FINISHED
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Starting attempt 1.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Started logcat recording.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Preparing device.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Installing apps.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Started crash detection.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Started crash monitoring.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Started performance monitoring.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Started video recording.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Starting instrumentation test.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Completed instrumentation test.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Stopped performance monitoring.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Stopped crash monitoring.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Stopped logcat recording.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Done. Test time = 155 (secs)
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Starting results processing. Attempt: 1
  7m 25s matrix-1a5cptg345w73 Pixel2-28 Completed results processing. Time taken = 10 (secs)
  7m 25s matrix-1a5cptg345w73 Pixel2-28 FINISHED
  7m 25s matrix-1a5cptg345w73 FINISHED
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Starting attempt 1.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Started logcat recording.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Preparing device.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Installing apps.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Retrieving Performance Environment information from the device.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Started crash detection.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Started crash monitoring.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Started performance monitoring.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Started video recording.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Starting instrumentation test.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Completed instrumentation test.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Stopped performance monitoring.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Stopped crash monitoring.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Stopped logcat recording.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Done. Test time = 162 (secs)
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Starting results processing. Attempt: 1
  7m 26s matrix-2khgqwh2j443k Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 26s matrix-2khgqwh2j443k Pixel2-28 FINISHED
  7m 26s matrix-2khgqwh2j443k FINISHED
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Starting attempt 1.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Started logcat recording.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Preparing device.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Installing apps.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Retrieving Performance Environment information from the device.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Started crash detection.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Started performance monitoring.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Started crash monitoring.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Started video recording.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Starting instrumentation test.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Completed instrumentation test.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Stopped performance monitoring.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Stopped crash monitoring.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Stopped logcat recording.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Done. Test time = 164 (secs)
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Starting results processing. Attempt: 1
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 26s matrix-2r7z3a7ysjjwj Pixel2-28 FINISHED
  7m 26s matrix-2r7z3a7ysjjwj FINISHED
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Starting attempt 1.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Started logcat recording.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Preparing device.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Installing apps.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Started crash detection.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Started performance monitoring.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Started crash monitoring.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Started video recording.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Starting instrumentation test.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Completed instrumentation test.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Stopped performance monitoring.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Stopped crash monitoring.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Stopped logcat recording.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Done. Test time = 188 (secs)
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Starting results processing. Attempt: 1
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 27s matrix-1qu5ibidneno1 Pixel2-28 FINISHED
  7m 27s matrix-1qu5ibidneno1 FINISHED
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Starting attempt 1.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Started logcat recording.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Preparing device.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Installing apps.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Started crash detection.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Started crash monitoring.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Started performance monitoring.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Started video recording.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Starting instrumentation test.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Completed instrumentation test.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Stopped performance monitoring.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Stopped crash monitoring.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Stopped logcat recording.
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Done. Test time = 146 (secs)
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Starting results processing. Attempt: 1
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 27s matrix-1gc32oa61qof2 Pixel2-28 FINISHED
  7m 27s matrix-1gc32oa61qof2 FINISHED
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Starting attempt 1.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Started logcat recording.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Preparing device.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Installing apps.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Started crash detection.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Started crash monitoring.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Started performance monitoring.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Started video recording.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Starting instrumentation test.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Completed instrumentation test.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Stopped performance monitoring.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Stopped crash monitoring.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Stopped logcat recording.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Done. Test time = 147 (secs)
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Starting results processing. Attempt: 1
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 28s matrix-2nhlibs17ch12 Pixel2-28 FINISHED
  7m 28s matrix-2nhlibs17ch12 FINISHED
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Starting attempt 1.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Started logcat recording.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Preparing device.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Installing apps.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Started crash detection.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Started crash monitoring.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Started performance monitoring.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Started video recording.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Starting instrumentation test.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Completed instrumentation test.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Stopped performance monitoring.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Stopped crash monitoring.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Stopped logcat recording.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Done. Test time = 148 (secs)
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Starting results processing. Attempt: 1
  7m 28s matrix-342s21w10b7b7 Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 28s matrix-342s21w10b7b7 Pixel2-28 FINISHED
  7m 28s matrix-342s21w10b7b7 FINISHED
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Starting attempt 1.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Started logcat recording.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Preparing device.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Installing apps.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Retrieving Performance Environment information from the device.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Started crash detection.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Started crash monitoring.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Started performance monitoring.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Started video recording.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Starting instrumentation test.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Completed instrumentation test.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Stopped performance monitoring.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Stopped crash monitoring.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Stopped logcat recording.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Done. Test time = 164 (secs)
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Starting results processing. Attempt: 1
  7m 28s matrix-78v4f0odf66sa Pixel2-28 Completed results processing. Time taken = 12 (secs)
  7m 28s matrix-78v4f0odf66sa Pixel2-28 FINISHED
  7m 28s matrix-78v4f0odf66sa FINISHED
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Starting attempt 1.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Started logcat recording.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Preparing device.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Installing apps.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Started crash detection.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Started crash monitoring.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Started performance monitoring.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Started video recording.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Starting instrumentation test.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Completed instrumentation test.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Stopped performance monitoring.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Stopped crash monitoring.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Stopped logcat recording.
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Done. Test time = 157 (secs)
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Starting results processing. Attempt: 1
  7m 29s matrix-30g2gl7vol143 Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 29s matrix-30g2gl7vol143 Pixel2-28 FINISHED
  7m 29s matrix-30g2gl7vol143 FINISHED
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Starting attempt 1.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Started logcat recording.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Preparing device.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Installing apps.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Retrieving Performance Environment information from the device.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Started crash detection.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Started performance monitoring.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Started crash monitoring.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Started video recording.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Starting instrumentation test.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Completed instrumentation test.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Stopped performance monitoring.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Stopped crash monitoring.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Stopped logcat recording.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Done. Test time = 196 (secs)
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Starting results processing. Attempt: 1
  7m 29s matrix-1d11pkrpw76et Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 29s matrix-1d11pkrpw76et Pixel2-28 FINISHED
  7m 29s matrix-1d11pkrpw76et FINISHED
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Starting attempt 1.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Started logcat recording.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Preparing device.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Installing apps.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Retrieving Performance Environment information from the device.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Started crash detection.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Started crash monitoring.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Started performance monitoring.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Started video recording.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Starting instrumentation test.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Completed instrumentation test.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Stopped performance monitoring.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Stopped crash monitoring.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Stopped logcat recording.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Done. Test time = 114 (secs)
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Starting results processing. Attempt: 1
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 29s matrix-10ze4j2hplr1g Pixel2-28 FINISHED
  7m 29s matrix-10ze4j2hplr1g FINISHED
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Starting attempt 1.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Started logcat recording.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Preparing device.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Installing apps.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Retrieving Performance Environment information from the device.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Started crash detection.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Started performance monitoring.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Started crash monitoring.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Started video recording.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Starting instrumentation test.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Completed instrumentation test.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Stopped performance monitoring.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Stopped crash monitoring.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Stopped logcat recording.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Done. Test time = 124 (secs)
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Starting results processing. Attempt: 1
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 30s matrix-33ktgnfvhm0yn Pixel2-28 FINISHED
  7m 30s matrix-33ktgnfvhm0yn FINISHED
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Starting attempt 1.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Started logcat recording.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Preparing device.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Installing apps.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Retrieving Performance Environment information from the device.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Started crash detection.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Started performance monitoring.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Started crash monitoring.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Started video recording.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Starting instrumentation test.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Completed instrumentation test.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Stopped performance monitoring.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Stopped crash monitoring.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Stopped logcat recording.
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Done. Test time = 126 (secs)
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Starting results processing. Attempt: 1
  7m 30s matrix-g1fihe7na35sa Pixel2-28 Completed results processing. Time taken = 10 (secs)
  7m 30s matrix-g1fihe7na35sa Pixel2-28 FINISHED
  7m 30s matrix-g1fihe7na35sa FINISHED
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Starting attempt 1.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Started logcat recording.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Preparing device.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Installing apps.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Started crash detection.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Started crash monitoring.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Started performance monitoring.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Started video recording.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Starting instrumentation test.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Completed instrumentation test.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Stopped performance monitoring.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Stopped crash monitoring.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Stopped logcat recording.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Done. Test time = 132 (secs)
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Starting results processing. Attempt: 1
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 30s matrix-2stz1ezd5rs57 Pixel2-28 FINISHED
  7m 30s matrix-2stz1ezd5rs57 FINISHED
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Starting attempt 1.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Started logcat recording.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Preparing device.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Installing apps.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Retrieving Performance Environment information from the device.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Started crash detection.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Started crash monitoring.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Started performance monitoring.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Started video recording.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Starting instrumentation test.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Completed instrumentation test.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Stopped performance monitoring.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Stopped crash monitoring.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Stopped logcat recording.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Done. Test time = 132 (secs)
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Starting results processing. Attempt: 1
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 31s matrix-3bdlybzdpiqm7 Pixel2-28 FINISHED
  7m 31s matrix-3bdlybzdpiqm7 FINISHED
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Starting attempt 1.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Started logcat recording.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Preparing device.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Installing apps.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Retrieving Performance Environment information from the device.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Started crash detection.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Started crash monitoring.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Started performance monitoring.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Started video recording.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Starting instrumentation test.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Completed instrumentation test.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Stopped performance monitoring.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Stopped crash monitoring.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Stopped logcat recording.
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Done. Test time = 126 (secs)
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Starting results processing. Attempt: 1
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 31s matrix-yq9xle35xtyaa Pixel2-28 FINISHED
  7m 31s matrix-yq9xle35xtyaa FINISHED
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Starting attempt 1.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Started logcat recording.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Preparing device.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Installing apps.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Retrieving Performance Environment information from the device.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Started crash detection.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Started performance monitoring.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Started crash monitoring.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Started video recording.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Starting instrumentation test.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Completed instrumentation test.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Stopped performance monitoring.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Stopped crash monitoring.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Stopped logcat recording.
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Done. Test time = 154 (secs)
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Starting results processing. Attempt: 1
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 Completed results processing. Time taken = 11 (secs)
  7m 32s matrix-3jszqfvdpo9ds Pixel2-28 FINISHED
  7m 32s matrix-3jszqfvdpo9ds FINISHED
  7m 32s matrix-2jazh0130r15a Pixel2-28 Starting attempt 1.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Started logcat recording.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Preparing device.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Installing apps.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Retrieving Pre-Test Package Stats information from the device.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Retrieving Performance Environment information from the device.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Started crash detection.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Started crash monitoring.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Started performance monitoring.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Started video recording.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Starting instrumentation test.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Completed instrumentation test.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Stopped performance monitoring.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Stopped crash monitoring.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Stopped logcat recording.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Retrieving Post-test Package Stats information from the device.
  7m 32s matrix-2jazh0130r15a Pixel2-28 Done. Test time = 146 (secs)
  7m 32s matrix-2jazh0130r15a Pixel2-28 Starting results processing. Attempt: 1
  7m 32s matrix-2jazh0130r15a Pixel2-28 Completed results processing. Time taken = 9 (secs)
  7m 32s matrix-2jazh0130r15a Pixel2-28 FINISHED
  7m 32s matrix-2jazh0130r15a FINISHED

FetchArtifacts
  .................................................
  Updating matrix file

CostReport
  Virtual devices
    $1.48 for 1h 29m

MatrixResultsReport
  30 / 30 (100.00%)

> Task :app:execFlank FAILED

    opened by bendb-instacart 9
  • Allow device configuration using fulladle

    Allow device configuration using fulladle

    Our multi-module project was previously using a per-module configuration for fladle. This was useful, and we did this before full-adle was created. It's a bit messy and also requires developers to remember to configure each module with fladle.

    One benefit of this approach however, was the ability to use to full suite of configuration options on a per-module basis. This includes the ability to decide with devices should be used for tests. This provides the ability to do things like run on physical devices for certain modules, but use emulators for others. Is there a way to do this with fulladle?

    Note: It would also be very useful to be able to configure if test-orchestrator should be used with the fulladleModuleConfig block.

    opened by asavill 0
  • Cost report is not being uploaded

    Cost report is not being uploaded

    Hi, I integrated fladle and configured it to run with a service account. when i run it i can see the matrix and results is uploaded to the bucket but i can't manage to get the Cost Reporting generated. how to generate the cost reporting like below ?

      Physical devices
    $0.08 for 1m

  Uploading [CostReport.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/...

    configuration:

    fladle {
    flankVersion = "22.+"
    serviceAccountCredentials = project.layout.projectDirectory.file("test-key.json")
    // Project Id is not needed if serviceAccountCredentials are set.
    // projectId("flank-gradle")
    useOrchestrator = false
    environmentVariables = [
            "clearPackageData": "true"
    ]
    testTargets = [
            "class com.example.test.TestFoo"
    ]
    devices = [
            ["model": "redfin", "version": "30"], // physical device
    ]
    localResultsDir = "results"
    smartFlankGcsPath = "gs://test-lab-etc/flank/test_app_android.xml"
    configs {
        instrumentationApk.set(project.provider { "${buildDir.toString()}/outputs/apk/androidTest/debugTesting/*.apk" })
        debugApk.set(project.provider { "${buildDir.toString()}/outputs/apk/debugTesting/*.apk" })
    }
    flakyTestAttempts = 1
    maxTestShards = 50
    filesToDownload = [
            '.*/sdcard/tempDir1/.*', '.*/data/local/tmp/tempDir2/.*','/sdcard/Download', '.*/sdcard/[^/]+\\.ec$'
    ]
    directoriesToPull = [
            '/sdcard/tempDir1', '/data/local/tmp/tempDir2','/sdcard/Download'
    ]
    testTimeout = "30m"
    resultsBucket = "test-lab-etc"
    outputStyle = 'multi'
    legacyJunitResult = false
    fullJunitResult = false
}

    output: Screen Shot 2022-09-01 at 10 43 55 PM

    opened by moetouban 6
  • dependOnAssemble does not work with Fulladle

    dependOnAssemble does not work with Fulladle

    Using dependOnAssemble with Fulladle tries to find TestedExtension in root project. TestedExtension is only part of Android libraries and application. Fulladle should ignore root module when dependOnAssemble is set to true.

    * What went wrong:
Could not determine the dependencies of task ':runFlank'.
> Could not create task ':execFlank'.
   > Could not find TestedExtension in logistics-rider-app-android

    Caused by: java.lang.IllegalArgumentException: Could not find TestedExtension in logistics-rider-app-android
        at com.osacky.flank.gradle.FladlePluginDelegate$createTasksForConfig$3.execute(FladlePluginDelegate.kt:125)
        at com.osacky.flank.gradle.FladlePluginDelegate$createTasksForConfig$3.execute(FladlePluginDelegate.kt:16)

    Fladle 0.17.3

    I checked the code and I thought of changi get method to find for TestedExtension to make it nullable and handle null case. If you guide me desired solution, I can open PR.

    opened by kozaxinan 0
  • Improve Compatibility with Workload Identity Federation

    Improve Compatibility with Workload Identity Federation

    When using Workload Identity Federation on GitHub Actions, there are a few compatibility issues that come up with Fladle. The token provided by the Google Auth GitHub Action doesn't include the Google Cloud Project name, which is a difference from creating a traditional long-lived service key.

    This causes a problem where Fladle fails, because the Google Cloud Project name cannot be found. The workaround is to provide the project name through an environment variable GOOGLE_CLOUD_PROJECT which can get picked up by Flank. An example of this is here https://github.com/zcash/secant-android-wallet/blob/45e14e6a4bcb0d5070ea17dc06feba7050f65ea6/.github/workflows/pull-request.yml#L337

    However this reveals another minor issue, which is that an error appears in the log like this:

    java.lang.NullPointerException: null cannot be cast to non-null type kotlin.String
Parsing /home/runner/work/secant-android-wallet/secant-android-wallet/gha-creds-e49e0.json failed:
kotlin.Unit
	at ftl.args.ArgsHelper.getProjectIdFromJson(ArgsHelper.kt:200)
	at ftl.args.ArgsHelper.fromUserProvidedCredentials(ArgsHelper.kt:195)
	at ftl.args.ArgsHelper.getUserProjectId(ArgsHelper.kt:188)
	at ftl.args.ArgsHelper.getDefaultProjectIdOrNull(ArgsHelper.kt:185)
	at ftl.config.common.CommonFlankConfig$Companion.default(CommonFlankConfig.kt:237)
	at ftl.config.CreateKt.defaultAndroidConfig(Create.kt:16)
	at ftl.domain.RunTestAndroidKt.invoke(RunTestAndroid.kt:47)
	at ftl.presentation.cli.firebase.test.android.AndroidRunCommand.run(AndroidRunCommand.kt:58)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:20[78](https://github.com/zcash/secant-android-wallet/runs/5586029293?check_suite_focus=true#step:6:78))
	at ftl.Main$main$1.invoke(Main.kt:12)
	at ftl.Main$main$1.invoke(Main.kt:10)
	at ftl.run.exception.ExceptionHandlerKt.withGlobalExceptionHandling(ExceptionHandler.kt:28)
	at ftl.run.exception.ExceptionHandlerKt.withGlobalExceptionHandling(ExceptionHandler.kt:17)
	at ftl.Main.main(Main.kt:10)

    So I'd like to suggest a few enhancements:

    1. Document the need to provide the environment variable when using Workload Identity Federation
    2. Improve the error message when the json key is missing the cloud project and there's no GOOGLE_CLOUD_PROJECT environment variable
    3. Don't print a stack trace to the log if the GOOGLE_CLOUD_PROJECT environment variable is present
    opened by ccjernigan 3
  • configureFulladle task is not compatible with configuration caching

    configureFulladle task is not compatible with configuration caching

    We just enabled configuration caching in our project, but had to disable it for our tests because of the following error:

    1 problem was found storing the configuration cache.
- Task `:configureFulladle` of type `org.gradle.api.DefaultTask`: cannot serialize object of type 'org.gradle.api.internal.project.DefaultProject', a subtype of 'org.gradle.api.Project', as these are not supported with the configuration cache.
  See https://docs.gradle.org/7.3/userguide/configuration_cache.html#config_cache:requirements:disallowed_types
    opened by grandstaish 10
  • Ability to identify flakey / re-run tests

    Ability to identify flakey / re-run tests

    Hi,

    When using the re-run feature, I wanted to know if there is a way to identify tests that have been re-run / are flakey. Is there an easy way to do this?

    Many Thanks

    opened by karan-jet 1
Owner
Nelson Osacky
Nelson Osacky
GitHub https://runningcode.github.io/fladle
3 types of Tests in Android (Unit - instrumentation - UI)

UnitTestingPractice 3 types of Tests in Android Unit instrumentation (Integration) UI Unit Testing benefits confirm code work like a charm simulate Ap

Ahmed Tawfiq 8 Mar 23, 2022
A custom instrumentation test runner for Android that generates XML reports for integration with other tools.

Android JUnit Report Test Runner Introduction The Android JUnit report test runner is a custom instrumentation test runner for Android that creates XM

Jason Sankey 148 Nov 25, 2022
Write Tests as Examples on the Method-under-test

Write Tests as Examples on the Method-under-test

Paul Methfessel 1 Apr 12, 2022
Raccoon is a lightweight response mocking framework that can be easily integrated into the Android UI tests.

Raccoon Medium Articles Checkout these article to get more insights about this library: How to integrate this in your Android Test Why Raccoon? There

Joseph James 52 Aug 15, 2022
Lbc-test-app - Test Android Senior Leboncoin

Test Android Senior Leboncoin ?? Mathieu EDET Overview Min API version : 24 This

null 0 Feb 7, 2022
Kotlin wrapper for React Test Renderer, which can be used to unit test React components in a Kotlin/JS project.

Kotlin API for React Test Renderer Kotlin wrapper for React Test Renderer, which can be used to unit test React components in a Kotlin/JS project. How

Xavier Cho 7 Jun 8, 2022
Toster - Small test dsl based on adb commands that allows you to test the mobile application close to user actions

toster Small test dsl based on adb commands that allows you to test the mobile a

Alexander Kulikovskiy 31 Sep 1, 2022
TestObserver to easily test LiveData and make assertions on them.

JCenter Update LiveData Testing is currently published on JCenter - it will serve packages until February 1st, 2022. LiveData Testing packages will be

Josef Raska 395 Dec 8, 2022
A simple project to help developers in writing their unit tests in Android Platform.

AndroidUnitTesting A simple project to help developers in writing their unit tests in Android Platform. This is not a multi-module project, but has th

Bruno Gabriel dos Santos 4 Nov 10, 2021
Espresso - Use Espresso to write concise, beautiful, and reliable Android UI tests

Espresso Use Espresso to write concise, beautiful, and reliable Android UI tests

Android For All 1 Jan 26, 2022
A collection of tests and easy to reuse pieces of code for bdk-jvm and bdk-android

Readme This repo is a collection of tests and easy to reuse pieces of code for bdk-jvm and bdk-android. Note that they don't aim to provide a full cov

thunderbiscuit 1 Jun 28, 2022
Most popular Mocking framework for unit tests written in Java

Most popular mocking framework for Java Current version is 3.x Still on Mockito 1.x? See what's new in Mockito 2! Mockito 3 does not introduce any bre

mockito 13.6k Jan 4, 2023
Using grpc-wiremock to mock responses in integrated tests of gRPC services

Utilizando grpc-wiremock para mockar respostas em testes integrados de serviços gRPC Este repositório possui um exemplo prático de como configurar e r

Tony A. Luz 2 Oct 28, 2021
A library that makes it easier to write high quality automated acceptance tests

Getting started with Serenity and Cucumber Serenity BDD is a library that makes it easier to write high quality automated acceptance tests, with power

ricardo larrahondo 1 Oct 20, 2021
A JUnit5 Platform TestEngine integrated with the official FHIR Validator to run profile and Questionnaire validation as tests.

?? FHIR Validator JUnit Engine A JUnit5 TestEngine to integrate the FHIR Validator into the JUnit5 ecosystem. Supports writing expected validation out

NAV IT 2 Feb 2, 2022
Automated tests using Rest-assured with Kotlin lang

Testes de API em Kotlin Pré-requisitos Instalar o Kotlin Ambiente Para executar os testes localmente, estou utilizando o ServeRest Link do Repo: https

Rafael Berçam 15 Dec 23, 2022
Setup CheckStyle, FindBugs, PMD and Lint for your Android project easily

android-quality-starter setup CheckStyle, FindBugs, PMD and Lint for your Android project easily This project adds gradle setup for quality tools ment

Piotr Wittchen 29 Sep 27, 2022
"Selenium for Android" (Test automate native or hybrid Android apps and the mobile web with Selendroid.) Join us on IRC #selendroid on freenode. Also confirm you have signed the CLA http://goo.gl/pAvxEI when making a Pull Request.

Selendroid Selendroid is a test automation framework which drives of the UI of Android native and hybrid applications (apps) and the mobile web with S

null 866 Dec 27, 2022
Linkester is an Android library that aims to help Android developers test their deep links implementation.

Linkester Linkester is an Android library that aims to help Android developers test their deep links implementation. The idea is to have a new launche

Ahmad Melegy 79 Dec 9, 2022