Feature request: Show Skipped Tests But Not Filtered Out Tests

[cwfitzgerald]

Motivation

wgpu uses nextest quite heavily. One particular quirk of the wgpu test suite is that we automatically run each one of our gpu-enabled tests on each gpu in the system. As we have tests for all sorts of features, not every gpu can run them. We currently make each test instance that cannot be run on that current system "no-op" itself. We use the test name to show why the test is disabled. Example of running a single test:

[INFO  xtask::test] Running cargo tests
warning: F:\Programming\wgpu\deno_webgpu\Cargo.toml: unused manifest key: target.cfg(not(target_arch = "wasm32")).dependencies.wgt.package
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.49s
    Starting 6 tests across 21 binaries (1511 skipped; run ID: b904e2de-06c3-49d6-9090-641819ae990e, nextest profile: default)
        PASS [   0.103s] wgpu-test::wgpu-test [Unsupported: Features] [Gl/NVIDIA GeForce RTX 4070/PCIe/SSE2/5] wgpu_test::binding_array::samplers::binding_array_samplers
        PASS [   0.648s] wgpu-test::wgpu-test [Executed] [Vulkan/AMD Radeon RX 7800 XT/1] wgpu_test::binding_array::samplers::binding_array_samplers
        PASS [   0.808s] wgpu-test::wgpu-test [Executed] [Vulkan/NVIDIA GeForce RTX 4070/0] wgpu_test::binding_array::samplers::binding_array_samplers                                                                                                                                                       
        PASS [   1.074s] wgpu-test::wgpu-test [Executed] [Dx12/Microsoft Basic Render Driver/4] wgpu_test::binding_array::samplers::binding_array_samplers                                                                                                                                                   
        PASS [   1.137s] wgpu-test::wgpu-test [Executed] [Dx12/NVIDIA GeForce RTX 4070/2] wgpu_test::binding_array::samplers::binding_array_samplers                                                                                                                                                         
        PASS [   1.189s] wgpu-test::wgpu-test [Executed] [Dx12/AMD Radeon RX 7800 XT/3] wgpu_test::binding_array::samplers::binding_array_samplers                                                                                                                                                           
------------
     Summary [   1.191s] 6 tests run: 6 passed, 1511 skipped

I would love for the "unsupported" test to show as something other than "PASS". I tried to do this by:

• Marking them as ignored in libtest-mock
• Having our xtask run with --status-level skip.

This worked great if I was running the whole suite. However, if I added any kind of filter like above, I would get 1500 lines of output instead of what I wanted, which was one.

Proposal

I would love some way to have a --status-level skip which includes ignored tests, but does not show filtered out tests.

Alternatives

I haven't really thought of any good ones. I would really love regex based coloring, or a more dataful contract between libtest-mock and nextest but those both sound much more complicated 😅

Additional context

No response

[sunshowers]

So there's a couple of things here.

The first is being able to mark a test as skipped at runtime. With libtest-mimic you can do that at list time, but with regular tests it really feels like there should be a way to exit out from a test at the beginning, and have that be clearly marked as distinct from PASS, but also not SKIP.

I'm not fully sure what to call this. But it's most similar to a list-time #[ignore] so let's call this "runtime-ignored".

---

The other consideration is having finer-grained indicators of test filtering. There's a few reasons we can choose to not run a test.

1. The test is marked as #[ignore] (reversed if --run-ignored=only is passed in)
2. The test is filtered out by the default filter.
3. The test doesn't match the provided string or expression filters (combining two of the variants here).
4. The test is in another partition.

There is also a ranking of how important these reasons are, from highest to lowest:

1. Marked as #[ignore]
2. String or expression mismatch
3. In another partition
4. Filtered out by the default filter

Here, "ranking" means "if a test was filtered out due to multiple causes and we needed to attribute it to one cause, which would it be?" I'm not 100% sure of this ranking, but it feels right.

Another consideration is signal-to-noise: how many tests do each of these skip? The smaller the number, the more relevant any information shown about them would be. It seems like:

• #[ignore] is generally rare, but conversely, --run-ignored=only will cause most tests to be skipped
• string/expression mismatches do a ton of filtering
• in-another-partition can do a lot of filtering in CI but less so for local runs
• default-filter is generally rare but not always (you can set up a profile to only run a small subset of tests).

Based on all this, I'm wondering if it makes sense to add a new filter with a level between pass and skip, covering both list-time and runtime ignores. But calling this filter "ignore" doesn't quite feel right, for two reasons:

1. With list-time ignores. you can invert the logic with --run-ignored=only. Saying "non-ignored tests are now ignored" is a bit confusing. (This is why I used the term "skip" rather than "ignore" originally.)
2. List-time and runtime ignores behave differently. List-time ignores can be executed, while runtime ignores cannot be.

There's also the asymmetry between the numbers of non-ignored and ignored tests here, which I think a good user experience would have to pay attention to.

I'd love your thoughts on this.

[sunshowers]

We could call this kind of runtime-determined inability to proceed "bail", maybe. It feels a bit distinct from #[ignore] because the latter is often used to (say) temporarily disable tests. #[ignore] is pretty overloaded and can mean lots of things unfortunately.

[cwfitzgerald]

With libtest-mimic you can do that at list time, but with regular tests it really feels like there should be a way to exit out from a test at the beginning, and have that be clearly marked as distinct from PASS, but also not SKIP.

Yeah that would be generally useful - in our case we are actually using libtest-mimic to do all of this, so not a big priority for us in particular, but having it categorized differently would be great.

Based on all this, I'm wondering if it makes sense to add a new filter with a level between pass and skip, covering both list-time and runtime ignores

Yeah this sounds good to me - ignore is a pretty special category on its own from the user's perspective and for a lot of users those tests might as well not exist. It might also be useful to have the ignored vs filter distinction in the count at the bottom of the screen. Like:

Summary [   0.224s] 5 tests run: 5 passed, 1 ignored, 1915 skipped

We could call this kind of runtime-determined inability to proceed "bail", maybe.

Bail sounds good to me!