Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docs(cloud): added documentation for retries #967

Merged
merged 5 commits into from
Aug 22, 2024
Merged

Docs(cloud): added documentation for retries #967

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fc12c0e
docs(cloud): added retries description
austyh Aug 18, 2024
f4765f2
docs(cloud): added retries description
austyh Aug 18, 2024
d78f5a1
docs(cloud): added retries description
austyh Aug 18, 2024
d9841c0
Apply suggestions from code review
austyh Aug 22, 2024
15e3989
Merge branch 'develop' into docs/retry
Malinskiy Aug 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/cloud/intro/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Whenever you submit an application and test application to Marathon Cloud, the f

When you run tests with Marathon Cloud for the first time, we begin storing the test history in our database.
The next time you run these tests, we already have information on the average time and the probability of a successful execution for each test.
Using this data, we calculate the necessary number of devices to ensure that your tests will complete within 15 minutes.
Using this data, we calculate the necessary number of devices to ensure that your tests will be completed within 15 minutes, be it 5 or 200.
We also monitor the progress of test executions and adjust the distribution of tests across devices as needed.
While the tests are running, our service can dynamically increase the number of devices to expedite the execution process.

Expand All @@ -35,6 +35,13 @@ If you prefer, you can enable the "isolated" parameter to manage device cleaning
but please note that this may lead to an increase in the number of devices required and the overall time of devices taken for testing.
However, the total execution time will still be 15 minutes.

### Retries
Retries are a cornerstone of effective testing. The larger your test suite, the greater the likelihood that one test might fail due to non-determinism, commonly known as test flakiness. While adding retries can improve test stability, it also increases the total testing time.
Marathon offers two retry strategies and several enhancements to strike a balance between stability and efficiency:
- Predictive retries: By analyzing past test executions, we can determine the success rate of each test. For flaky tests, we can run multiple executions to ensure at least one succeeds.
- Common retries: If a test fails, we will rerun it up to three times. Marathon will attempt to run the test on a different device to minimize the environmental impact on the results.
- Additional features: Sorting, batching, test rebalancing, and more.




Expand Down
Loading
Loading