Fix nightly test runs. #2663
Fix nightly test runs. #2663
Conversation
| --directories-to-pull=/sdcard \ | ||
| --results-dir=${GCLOUD_RESULTS_DIR} \ | ||
| --results-history-name=${{ inputs.lib }} \ | ||
| --timeout=20m --no-auto-google-login --no-record-video --no-performance-metrics --num-flaky-test-attempts=1 || true |
There was a problem hiding this comment.
The || true is a little trick to ensure the test run does not return a non-zero value to the shell, since that will cause the entire step to end before we have looped through all API levels.
This is fine because the "Test Report" step below fails if any tests fail.
| - name: Copy Test Results | ||
| continue-on-error: true |
There was a problem hiding this comment.
I put this here and on the "Run Tests" step just to help ensure we actually loop though all API levels.
| if $IS_PR ; then | ||
| echo "No test results found" | ||
| exit 1 |
There was a problem hiding this comment.
Error on PR to make sure expected tests were actually run.
| # Most used according to https://gs.statcounter.com/android-version-market-share/mobile-tablet/worldwide | ||
| PR_API_VERSION: "34" | ||
| FULL_API_RANGE: "28 29 30 31 32 33 34 35" |
There was a problem hiding this comment.
As I have said on previous PR's but wow, how does Github Actions not support YAML Anchors???
I am actively watching the (4 year old!) issue requesting the feature which may actually be getting fulfilled soon, but I will specially mention this needs to be updated in multiple places on out Github Actions doc.
This PR changes the Nightly test structure from running one Firebase Test Lab with all supported API levels simultaneously to running them sequentially.
If we try to run 8 instances (API 28 to 35) of our tests at once, matrixes across all of our libraries, it is simply too many simultaneous API calls to the server and the tokens gets invalidated. This also probably would have lead to a lot of flapping tests in MobileSync when manipulating data on the server.
Here is an example of the report from a test I ran with just 28 and 35:
In production it will show all (8 as of now) supported API levels in a single report page per library: