Force test run on a single device

[nikolas-scaleforce]

Is your feature request related to a problem? Please describe.
We are running multiple different test batches on the same CI pipeline. Initially we start a few emulator instances and run the tests that allow for parallel execution. The last test batch is for sequential tests that functionally do not support being run in parallel - but they still run in parallel because we have the multiple emulator instances already started.

Describe the solution you'd like
We need a way to force Marathon to run the tests against a single emulator/device without having to explicitly kill(disconnect) all running emulators/devices for this to work.

Describe alternatives you've considered
Alternative is mentioned above - go over the running emulators and kill all but one, but this makes the pipeline setup more complex

Additional context
We use OSS runner with the Gradle plugin and all test batches have their own annotation filter applied, so we use different commands to start the tests like ./gradlew marathonStandardDebugAndroidTest -PmarathonAnnotationFilters=Sequential. I looked over the documentation multiple times but I couldn't find an existing option to do this

[Malinskiy]

All the tests are expected to be homogeneous in terms of running strategy, this has always been an assumption test runs in marathon. If you have a list of tests that require different constraints - I suggest to start a separate test run and connect as many or as few devices as needed directly via adb or by splitting your devices into groups combined under different adb servers and pointing marathon to an appropriate adb server

I'm happy to consider alternative technical proposals, but I don't see any value from overcomplicating an already complex test run logic when simply running a different test run would achieve exactly the same.

FYI for clarity it's probably best to not call separate test runs as batches because in marathon a batch is a unique set of tests

I don't have the code for marathonAnnotationFilters property, but I assume it dynamically populates marathonfile via gradle configuration. This is the expected way to group tests for execution in gradle plugin. For CLI usage a single Marathonfile and multiple filter files is usually the preferred method https://docs.marathonlabs.io/runner/configuration/filtering#values-file-filtering

[nikolas-scaleforce]

You got it exactly right, the property uses JUnit CustomFilter to populate the marathon filters via gradle configuration dynamically for tests with different constraints in batches, suites, sets of tests or whatever you find suitable to call them - sorry if I didn't use the correct terminology regarding Marathon in particular :)

I totally get what you're saying but I find a problem with the assumption about the control over the devices - not everyone has their own dedicated device/emulator farm with full control over the way devices are being spun up/connected to the machine executing the tests. In current times many people use some kind of cloud service to run application builds and tests. In our case we use Azure VM agents - emulator setup and boot is a slow and expensive operation even for a single pipeline, let alone having separate test runs. Of course I could still spin up and down emulators depending on the current set of tests being run but this overcomplicates and slows down the overall test job execution. Considering the improvements I get from running the tests in parallel it is kind of acceptable, I was just wondering if there is a way to target just one device in the case where a specific set of tests has such constraints that allows them to run only sequentially.

Anyway, thanks for the quick reply! Apart from this "limitation" in my scenario, I want to say that this is a great tool, much better than anything else I've tried in terms of performance and flexibility, either OSS or Google provided. Amazing job 🚀

[Malinskiy]

That's okay regarding the terminology. It's just sometimes hard to understand if we're talking about the same thing, so I needed to clarify it.

I understand the constraint on the device control. Marathon is a large piece of the puzzle in terms of running tests at scale, but it's not a full solution. Doing that requires tighter coupling between marathon and the device provisioning logic as well as all the other pieces at play here. I tried building the full solution here in Marathon, but lots of users have very different and diverging pieces around Marathon. That's why I've separated solving the whole problem domain into a commercial setting rather than open-source: tight coupling would mean complicated expertise required to support it as well as less flexibility between components. Marathon OSS on the other hand is as flexible as possible on the other hand. As I've replied before, feel free to suggest a technical design for supporting unparallelizable tests.