Feature request: Ability to run tests from main thread

[mdevils]

Motivation

I'm the developer of osakit crate. That crate provides an API to work with OSAKit Framework from MacOS.

The problem is that OSAKit only works properly when executed from the main thread.

nextest runs tests from a separate thread even if --test-threads is set to 1. This makes OSAKit tests fail.

The same problem also applies basically to any rust application which uses MacOS UI Frameworks, because they also require main thread.

Proposal

I suggest adding a CLI option which will make nextest run tests in the main thread only.

Alternatives

This question was asked for cargo test before:

1. https://stackoverflow.com/questions/43458194/is-there-any-way-to-tell-cargo-to-run-its-tests-on-the-main-thread
2. Ability to run cargo tests on main thread. rust-lang/rust#128061
3. Segfault when using cargo test with glfw rust-lang/cargo#3927
4. Cargo test makes gtk application crash on OSX rust-lang/cargo#5438

Additional context

No response

[sunshowers]

Hi — thanks for the issue! I agree this is unfortunate, when first building nextest I thought that if you run a single test, libtest wouldn't create a thread for it even if --test-threads=1 isn't specified.

If I were building nextest from scratch today, I'd definitely always pass in --test-threads=1. But I'm worried about backwards compatibility if we just change the default here, for two reasons: custom test harnesses which may not support --test-threads, and tests that depend on being run in another thread.

I think what we can do is make this an option to opt into, though. I'm torn between a purpose-built option for this and a general "pass these additional arguments into the test binary while running tests" option. What do you think?

[mdevils]

Hello @sunshowers! Thank you for your response.

Yeah, I totally understand the backwards compatibility issue. I'd keep --test-threads=1 as is and would introduce a new option.

Could you please describe the options you are describing in more details?

general "pass these additional arguments into the test binary while running tests" option

Is this what happens when I run just normal cargo test and then add more arguments after --? If so, I need to understand how I can use it without writing my own test runner and keeping the tests more or less the same as they are right now.

[sunshowers]

What I was thinking was, in configuration, you could say:

[profile.default]
run-on-main-thread = true
# or: run-args = ["--test-threads=1"]

And then nextest would add --test-threads=1 to the end of every test that's run.

Yeah, it's what happens when you run cargo test -- --test-threads=1 -- the arguments get passed into the test binary.

[mdevils]

[profile.default]
run-on-main-thread = true

This looks great to me. I think it's more like a project configuration rather than an argument for a particular run. Because it can be critical for the tests to succeed. Without them running in the main thread they just fail and it would be a false negative.

[sunshowers]

Yeah, agreed -- it should go in configuration because it's an inherent property of the test, not of a particular invocation of nextest.

[sunshowers]

Hmm, so I have the option working, but it looks like it won't help with newer versions of Rust, which always create a thread to run the test in, even if --test-threads=1 is specified: rust-lang/rust#104053

(My memory of this was from back in 2022 when nextest first came out, and was outdated for that reason :) )

Nextest just delegates to libtest here, so I don't think nextest can do anything special unfortunately. You may be able to use a custom test harness to achieve this.

[sunshowers]

OK, based on my read, it looks like with a combination of two things it should be possible to run tests on the main thread:

• using libtest-mimic, see https://github.com/LukasKalbertodt/libtest-mimic/blob/63e4ec13f15c4adc5b36e3cc0c5f49e19b50f480/src/lib.rs#L498-L510
• having nextest be able to pass --test-threads=1 to libtest-mimic.

Can you confirm that using libtest-mimic is an option for you? If so then I can add an extra-args feature to nextest (I don't want to call that run-on-main-thread because this is quite runner-specific -- libtest-mimic does a different thing from libtest.)

[mdevils]

@sunshowers I will check if I using libtest-mimic does the job.

[eugenesvk]

Just FYI I've run the failing test spawning the original issue with libtest-mimic and --test-threads=1, and it doesn't fail anymore! Thanks for the tip

[sunshowers]

Thanks, have put up #1968, which lets you specify extra arguments on a per-test (or global) basis. Hoping to get this out in a beta release soon -- would love it if you gave it a shot!

[sunshowers]

ok, cargo-nextest 0.9.86-b.3 has support for this. Could you try it out?

cargo nextest self update --version 0.9.86-b.3

Then, follow the instructions at https://nexte.st/docs/configuration/extra-args.

(Let me know if the instructions seem incorrect.)

[mdevils]

@sunshowers I've tried 0.9.86-b.3 and it works great! Thank you for fast fix. Also thank you for recommending to use libtest-mimic. I've also created a crate libtest-mimic-collect to serve as a drop-in replacement to keep tests as-is while changing the test runner.

[sunshowers]

Thanks. Note that depending on the implementation, beforeAll/afterAll might not work for nextest. Nextest does support setup scripts: https://nexte.st/docs/configuration/setup-scripts/ -- not teardown scripts yet, but that's on the roadmap.

beforeEach/afterEach can be done in libtest-mimic I'm quite sure -- you just need to check that the list argument hasn't been passed in.

[mdevils]

Could you please hint me how do I find beforeEach/afterEach in libtest-mimic?

[sunshowers]

Assuming you're using nextest which runs each test in a separate process, beforeEach can just be done at the beginning of main before calling into libtest-mimic.

afterEach you can probably do after calling into libtest-mimic. Since you're running tests on the main thread, you may have to catch_unwind the call into libtest-mimic. You can also use a scope guard to ensure that your cleanup happens even in the face of panics.

[mdevils]

What you're describing sounds more like what beforeAll() and afterAll() do in jest / mocha / other JS-frameworks. beforeEach() should run before each individual test case inside a suite (mod) and afterEach() after each test.

Here is the some rust metacode example:

mod test {
  fn before_all() { /* ... */ }
  fn before_each() { /* ... */ }
  fn my_test() { /* ... */ }
  fn after_each() { /* ... */ }
  fn after_all() { /* ... */ }

  mod sub {
    fn before_all() { /* ... */ }
    fn before_each() { /* ... */ }
    fn my_other_test_n1() { /* ... */ }
    fn my_other_test_n2() { /* ... */ }
    fn after_each() { /* ... */ }
    fn after_all() { /* ... */ }
  }
}

The desired behaviour when I run this structure would be (for simplicity let's say it's a single threaded execution):

test::before_all();
  test::before_each();
    test::my_test();
  test::after_each();
  test::sub::before_all();
    test::before_each();
      test::sub::before_each();
        test::sub::my_other_test_n1();
      test::sub::after_each();
    test::after_each();
    test::before_each();
      test::sub::before_each();
        test::sub::my_other_test_n2();
      test::sub::after_each();
    test::after_each();
  test::sub::after_all();
test::after_all();

Basically that's the behavior which comes by default with popular JS frameworks such as jest / mocha and many others. I need something like that in rust.

[sunshowers]

Ah, I see you need a representation of subtests. Yes, that wouldn't directly be possible -- by beforeEach, I meant running before each test, not necessarily before each subtest.

You can do this kind of subtest before/after code management by hand (e.g. use a callback, call the functions directly, or set up a context for each subtest where the constructor does whatever beforeEach would do). I think the Rust community tends to shy away from this kind of global before/after pattern because of how indirect it is.

What nextest would give you though, is each test runs in a separate process, in parallel -- so you get the performance benefits of parallel execution while running each test in the main thread of its corresponding process.

[sunshowers]

Going to close this discussion since the need has been addressed via https://nexte.st/docs/configuration/extra-args/. Thanks for the report!

[mdevils]

Sure, thanks again!