[feature request]: Add ability to throttle RAM usage

[appetrosyan]

Feature summary

Add the ability to reduce the number of running tests when one test is likely to take over the entirety of the available RAM.

Essentially cargo nextest run -j 1 except smart.

Motivation

I'm dealing with large Rust projects. It used to be that 32GiB was enough, but these days, a single test run on a 16-core machine can easily gobble all of the resident RAM. In most cases this is preventable by reducing the number of concurrent tests being run, which in case of nextest is still often justified, because that still lets me collect flakyness and other statistics. But it kinda slows a big project to a crawl.

Given that nextest can control how many tests it can spawn, and sometimes kill them too, I was wondering if it were possible to do one (or all) of the following:

1. Kill the test that exceeds X amount of RAM, push it into a queue, and run it at the end. If there's enough available RAM then.
2. When all running tests exceed a certain RAM threshold, do not spawn new tests until the used RAM drops below a given threshold.
3. Allow the user to mark tests as RAM-intensive, so that they could be run last, or with fewer concurrent tests, or at least, to prevent a group of them running at the same time, and causing the entire test run to get killed by systemd-oomd.
4. Bail before the nextest is killed by oomd, and instead show the user that the tests are too RAM intensive.

[sunshowers]

Thanks for the request! Definitely agree that this is suboptimal. Backing off under memory (or I/O) pressure is definitely one of the things we've considered to do in nextest, but hasn't been implemented yet. There's a related idea to use a small DSL to turn -j from just a static number into something that can take memory and CPU into account.

Allow the user to mark tests as RAM-intensive, so that they could be run last, or with fewer concurrent tests, or at least, to prevent a group of them running at the same time, and causing the entire test run to get killed by systemd-oomd.

This can be done via heavy tests or test groups.

[appetrosyan]

This can be done via heavy tests or test groups.

So, essentially extend the heavy-test mechanism.

The reason it needs extension is that while it could be used to reduce the number of tests that run concurrently, this does not stop the RAM for other tests from being allocated, if e.g. one person on a team has 16 logical CPUs and marks a test as requiring 16, and another person with the same amount of ram, but 32 logical CPUs runs that test.

Test groups seem much more amenable to what I'm trying to accomplish, and is an appropriate workaround.

---

Regardless, an automatic solution would probably also be useful. Would you be open to me implementing this?

[sunshowers]

Thanks for the offer! What do you have in mind as an implementation strategy?

[appetrosyan]

Simply put: monitor RAM usage, if it reaches a certain threshold stop spawning new tests until it drops below a certain threshold. Then I'll simply add another optional field to the configuration which basically states that all tests in this group can use X amount of RAM, which combined with the initial reading, will avoid tests being killed by OOM killer.

Then we can get fancy, and do a lite version of the knapsack problem to choose tests that could be run with the given RAM and given current readings.

Ideally, there can also be a switch which measures the RAM used if it's above a significant percentage of the total, e.g. THRESHOLD == TOTAL_RAM / TOTAL_CORES * THREADS_USED_IN_TEST, so you have a convenient automatic switch that marks all RAM-intensive tests just as long-running tests are marked, except it can be saved into a file, and used in .config/nextest.toml.

[appetrosyan]

So I suppose nextest-filtering is my first stop.