How to write independent test suites with nextest

[Nugine]

I'd like to test different web services which use the same application-level protocol.

So I plan to write a reusable e2e test suite.

• distributes a single binary independent from Rust toolchain
• inputs configurations (host:port, test filters, ...)
• outputs test summary and results
• runs with awesome features from nextest

It will be like a custom test harness with nextest runner, packed in the same binary.

My questions:

• Is it possible?
• How to implement?

[sunshowers]

Thanks for the great question! This is honestly not a use case I'd thought about before.

I think the use case makes sense but I'm worried it's a lot of work. Currently, cargo-nextest (including nextest-runner) is really tied to Cargo and rustc. It uses Cargo to discover binaries, to figure out what environment variables to set, and for a number of other things.

This is not inherent to nextest's model: you could certainly imagine the runner working against a more abstract test discovery and running interface. I think doing so is probably going to make nextest's code better as well, and also expose a bunch of hidden dependencies to Cargo. But in order to achieve this you'd probably need to roll up your sleeves and do the design and implementation work.

This would also likely enable nextest to work against non-Cargo build systems like Buck and Bazel, something that's been desired for a while: #1790. And also likely against non-Rust languages as well. It essentially decouples nextest's features from their current reliance on Cargo and rustc.

To set expectations: I don't anticipate getting to this any time soon. Maybe in a world where we start using Buck2 at my employer, but not any time soon. But I think it would be a good idea if you'd like to try and flesh out a design for this using traits.

[Nugine]

Basically what I'm looking for is a full-featured embedded test framework (custom harness + runner).
It's like gtest and catch2 in C++ world. I think there is an empty place in Rust ecosystem and nextest seems to be the closest.

[sunshowers]

@Nugine Gotcha! That is a fascinating use case. I will be honest this would be a bunch of work -- nextest is not currently built to be consumed as a library in this regard. But I'm not opposed to it and I think it would be valuable.

One consideration is how to handle nextest's process management code. Currently many nextest features like timeouts depend on running each test in its own process -- in an embedded world, there would need to be some kind of in-process cancellation protocol. That would likely be a lot of design and implementation work.

[NobodyXu]

in an embedded world, there would need to be some kind of in-process cancellation protocol. That would likely be a lot of design and implementation work.

I think using pthread_kill combined with pthread_exit would at least work on any pthread env?

A even crazier idea would be to run everything within miri, so you could kill the test at anytime, though it's normally slower than running native code.

[sunshowers]

I believe thread killing has a long history of not really working out too well. For example my guess is that locked mutexes don't get marked poisoned via that route, even though the reasoning for doing so is the same as doing that for panics.

[NobodyXu]

That's true, I guess stuff like miri/wasm is actually a better bet than using thread + signal.