"--list-tests" should respect the selected profile and settings by "--test"

[matsduf]

zonemaster-cli --list-tests lists all test modules and all test cases in each test module.

• It accepts that a --profile=PROFILE is added to the command, but the selected profile is ignored.
• It also accepts a --test=EXPR setting (based on More flexible --test option #359) but it does not respect that setting, i.e. with "--test=basic" only tests in Basic module will be run, but zonemaster-cli --list-tests --test=basic will still list all defined test cases.

I want to have answer to the following question:

• What test cases will be run if I use a specific profile and "--test" option?

I suggest that disabled test caes are marked as such, e.g. the following output:

$ zonemaster-cli --profile special-profile.json --list-tests
Address
       address01
       address02
       address03 (disabled)

Basic
         basic01
         basic02
         basic03

Connectivity
  connectivity01
  connectivity02
  connectivity03 (disabled)
  connectivity04 (disabled)

Consistency
   consistency01
   consistency02
   consistency03
   consistency04
   consistency05
   consistency06

DNSSEC
        dnssec01 (disabled)
        dnssec02 (disabled)
        dnssec03 (disabled)
        dnssec04 (disabled)
        dnssec05 (disabled)
        dnssec06 (disabled)
        dnssec07 (disabled)
        dnssec08 (disabled)
        dnssec09 (disabled)
        dnssec10 (disabled)
        dnssec11 (disabled)
        dnssec13 (disabled)
        dnssec14 (disabled)
        dnssec15 (disabled)
        dnssec16 (disabled)
        dnssec17 (disabled)
        dnssec18 (disabled)

Delegation
    delegation01 (disabled)
    delegation02 (disabled)
    delegation03 (disabled)
    delegation04 (disabled)
    delegation05 (disabled)
    delegation06 (disabled)
    delegation07 (disabled)

Nameserver
    nameserver01
    nameserver02
    nameserver03
    nameserver04
    nameserver05
    nameserver06
    nameserver07
    nameserver08
    nameserver09
    nameserver10
    nameserver11
    nameserver12
    nameserver13
    nameserver15

Syntax
        syntax01
        syntax02
        syntax03
        syntax04
        syntax05
        syntax06
        syntax07
        syntax08

Zone
          zone01
          zone02
          zone03
          zone04
          zone05
          zone06
          zone07
          zone08
          zone09
          zone10
          zone11

[mattias-p]

For context, it's possible to use --dump-profile to query:

• the full set of test cases, as well as
• the subset of test cases enabled in a custom profile.

zonemaster-cli --dump-profile | jq .test_cases
zonemaster-cli --dump-profile --profile custom-profile.json | jq .test_cases

A notable difference from the --list-tests feature is that --dump-profile only outputs the names of test cases, not test modules.

[matsduf]

That could be a work-around, but that is not documented, and it does not handle settings by "--test" (see updated description above). And it is more complex for the users.

[mattias-p]

that is not documented

That could be fixed.

it does not handle settings by "--test"

It will once #359 is merged.

And it is more complex for the users.

We should definitely keep the --list-tests option, and I agree that it makes sense to make it respect --profile and --test.

[mattias-p]

I have two remarks on the current suggestion in the issue description.

1. I think test modules that don't have any enabled test cases should be considered disabled themselves.
2. Why not omit disabled tests from the output entirely? Do you have a use case in mind where you specifically need to see all the test cases and which ones are disabled?

[matsduf]

1. I think test modules that don't have any enabled test cases should be considered disabled themselves.

That could be fine.

2. Why not omit disabled tests from the output entirely? Do you have a use case in mind where you specifically need to see all the test cases and which ones are disabled?

That could be fine too. If both --profile and --test are respected then zonemaster-cli --test=all --list-tests would output all test cases. Most import is to be able to check what test cases that will be run with a certain options. Keeping the disabled test cases in the list is not important.

[matsduf]

it does not handle settings by "--test"

It will once #359 is merged.

If #359 is updated. As it works now zonemaster-cli --list-tests --test basic lists all test cases.

[mattias-p]

it does not handle settings by "--test"

It will once #359 is merged.

If #359 is updated. As it works now zonemaster-cli --list-tests --test basic lists all test cases.

Actually, --dump-profile already respects --test in #359 without further updates.

Why not omit disabled tests from the output entirely? Do you have a use case in mind where you specifically need to see all the test cases and which ones are disabled?

If both --profile and --test are respected then zonemaster-cli --test=all --list-tests would output all test cases. Most import is to be able to check what test cases that will be run with a certain options. Keeping the disabled test cases in the list is not important.

Thanks for clarifying.

I feel really good about having --list-tests respect --profile and --test, and even better about also having it omit disabled test cases and test modules with only disabled test cases.

[matsduf]

I feel really good about having --list-tests respect --profile and --test, and even better about also having it omit disabled test cases and test modules with only disabled test cases.

I think it will be enough to include the test cases, and not the test modules.

[mattias-p]

I feel really good about having --list-tests respect --profile and --test, and even better about also having it omit disabled test cases and test modules with only disabled test cases.

I think it will be enough to include the test cases, and not the test modules.

Yeah, maybe. That certainly has its merits.