Proptesting CLI apps

[killercup]

@Centril and I talked a bit about writing property tests for CLI apps. The general idea is to write input generation and assertions on CLI app invocations without much overhead (e.g. without having to deal with manually formatting types as CLI arguments).

The most trivial implementation that already works is probably this:

proptest! {
    #[test]
    fn same_input_back(string: String) {
        assert_cli::Assert::command(&["echo", &string])
            .stdout().is(&string)
            .unwrap();
    }
}

It generates random strings and asserts that echos gives us back the same string it was fed with.

We found that aside from the prop-testing specific challenges, there are also CLI-specific helpers needed, e.g. to easily generate temp dirs with (also generated) files, and that OS-specific or environment specific settings need to be able to be passed as arguments (or to be overwritten to reduce/eliminate dependence on global state).

In general, we think that we can use assert_cli to write these helpers. What we need to find is a way to express relations between arguments (inputs) and the asserts (like "stdout contains").

---

For the record but not meant as a concrete suggestions as something to implement today, I proposed a macro/DLS like:

cli_check!(cargo_binary("brighten"), ("--color": Color, "--amount": 0f32..=1) -> (stdout: Color));

that passes two arguments to a binary called brighten and asserts that, given valid arguments, the tool writes a string to stdout that can be parsed as Color. Obviously missing here is a way to setup the environment. Side-effect of this syntax is that generating invalid (or omitting) arguments and asserting that the program fails is easy.

[Centril]

Some more assorted notes of the top of my head:

Temp dirs

When we generating temporary directories, there are a few aspects one might want to control:

• The depth of the recursive (nested) directory structure
• The number of files per directory, possibly growing or decreasing as the depth grows
• The type of files, and their file endings
• The content of the files.

It seems to me that providing a bunch of Strategy combinators could satisfy this wealth of needed configuration. Generation of file contents also could use the standard regex based string generation or if a more rich grammar is needed, you could use CFGs (see: proptest-rs/proptest#61).

Extra: Interactive CLI programs

Some CLI programs are interactive; for those kinds of programs, you not only have input/output, but the requested inputs and requested outputs can depend on previous input/output pairs. Sometimes multiple inputs are also requested based on previous inputs. This can probably be modeled with State Machines (see: proptest-rs/proptest#28).

In my BSc thesis (see: http://publications.lib.chalmers.se/records/fulltext/251311/251311.pdf, repo: https://github.com/DATX02-17-26/DATX02-17-26), we discuss a Haskell DSL that is related to this (see the testing bit, the other parts are irrelevant for CLI apps).

Extra: Testing protocols

For testing protocols, which is tangentially related to CLI apps, there's the MSc thesis "Bidirectional Testing of Communicating Systems" which provides the tool called SessionCheck (see: http://publications.lib.chalmers.se/records/fulltext/254897/254897.pdf, repo: https://github.com/MaximilianAlgehed/SessionCheck)

cc @MaximilianAlgehed on the feasibility of reusing ideas from SessionCheck here.

[epage]

Love the idea and would like to help with whatever we come up with but also want to bring up some concerns.

Also, in case people aren't aware, we have both assert_cmd and assert_fs that we can leverage.

Some challenges

• Performance: end-to-end testing tends to be slower, so doing prop testing at this level (where N iterations are run per test) might be a bit intense
• Variability: I'm a bit unsure in the ability to express relations beyond what assert_cmd can already do.
• File/dir strategies: This only works if there is little required structure. Otherwise, I feel like you'll have to just use multiple string strategies for what is exactly under test and manually construct the tempdir directly.
  • I'm immediately thinking of cobalt in being a static site generator.
  • Even cargo-generate needs some structure (passing to stdin newline delimited strings where each would have its own strategy).
    • Oh, state machine proptesting? Neat

btw for interactivity, we might want to explore how we can reuse or be inspired by rexpect

[MaximilianAlgehed]

@Centril I'm no rust expert, but I don't think it matters here.
I'm confident the ideas from SessionCheck carry over at least somewhat to CLI applications.
The big problem with SessionCheck is that it is essentially message-oriented, meaning that the I/O of the CLI applications needs to be discretized somehow. For applications, for example REPL-style applications this is of course no problem what so ever.

But I'm more curious why you need the bidirectionality which SessionCheck offers, have you got two programs interacting using the CLI?

Best,
Max

[BurntSushi]

If y'all need examples, then xsv has been using quickcheck for this purpose. To address performance concerns, tests are compiled with opt-level = 3.