Upcoming behavior change: produce a non-zero exit code (4) if no tests were run

[sunshowers]

Motivation

All but the simplest of projects have at least one test, and expect each invocation of nextest to run at least one test. Running zero tests likely indicates some kind of configuration issue.

Note that this is not a real error, so we're choosing to use the advisory exit code 4. Exit code 1 is for unexpected errors, 2 is for parsing errors, and 3 often has a specific meaning as well. 4 is the smallest unreserved code and fits this use case well.

Per our stability policy, this counts as a behavior change and will follow the rules outlined there, with a 3 month period. I don't anticipate any particular issues here -- some CI might break but it'll be correctly so.

Note that the behavior change is only for cargo nextest run, not list.

Update 2024-08-19

Based on feedback, we're going to make this a CLI option called --no-tests with three possible values:

• pass: same behavior as nextest 0.9.72 and below
• warn: new behavior, temporary default
• fail: new behavior, future default

The --no-tests=warn and pass is meant for cases like cargo hack --feature-powerset nextest run, where some combinations of features can lead to no tests being run in an expected manner.

See #1648.

Timeline

• 2024-08-18: Start warning about this with cargo-nextest 0.9.74
• After 2024-11-18: change behavior.