Feature idea: Condition for retrying tests

[torokati44]

We're considering switching to nextest to run our Rust tests in CI over at Ruffle. (Cc: ruffle-rs/ruffle#14461) It's been great so far!
The final issue we have is, that compared to vanilla cargo test - perhaps because of the many processes starting and stopping - we are occasionally triggering what looks like a rare bug in the Ubuntu 22.04 graphics drivers.
See for example: https://github.com/torokati44/ruffle/actions/runs/7254876905/job/19764476196#step:12:1448

We wouldn't want to allow retries/flakyness in general just because of this, because we don't actually have flaky tests, it's only the environment that's not ideal. And we would like to know if one of our tests actually becomes flaky, by having it fail.

Rather, what I think would be great for this, is if there was a way to say that we want to retry tests only if they failed because of this specific reason, but not otherwise.

For example, filtering conditions for this could be the exit code, or a substring/tail of the stdout/stderr output of the testing process.

[sunshowers]

Seems reasonable. I'd prefer to gate this only on exit codes to avoid heuristics that can go wrong. Would that be good enough for you?

[torokati44]

Would that be good enough for you?

Since in this case, the test gets a SIGABRT, which almost certainly results in an exit code of 134 - at least on Linux, where our concrete problem at hand manifests -, and this exit code is not likely to be used for anything else, I think so, yeah. :)

[sunshowers]

I have the execution side of this mostly working -- I started writing a parser for this but realized that it'll likely conflict with some other work going on in #1159. Hopefully we'll figure out a way to sequence this work soon.

[torokati44]

Oh, the mentioned PR has landed now! ^^