Ducktape current state - July 2022

[stan-is-hate]

We've released a lot of improvements across all of the branches over the recent months (and years), some small ones and some quite large. This post is an attempt to summarize at least a part of them.

• there's been a ton of improvements around how ducktape handles errors in test runner processes and general stability:
  • we're handling exceptions when reporting other exceptions better,
  • as well as exceptions in the main run loop, and ensure no zombie processes hang up after something goes wrong.
  • There's still a lot of work to do there, but already there's been a lot of work focused on internal ducktape stability and reliability
• added a feature to run arbitrary "checker" code on ssh errors - our use case for it was to check aws api for whether the ec2 instance was still alive or not, but any kind of check can be implemented
• ducktape now generates a helpful output that lists all failed tests - you can copy-paste it into the next execution. It will also generate a test suite file with the same info
• ducktape has a --deflake N flag now - if provided, it will attempt to rerun each failed test up to N times, and if it eventually passes, it will be marked as Flaky
• html output will now list all the actual exception for each failed test - you no longer need to download tgz just to see why the test failed
• ducktape will now print a warning if a test declares more nodes than it actually needs - and an option to turn that warning into an error
• and we've added a small feature to print how many total nodes are needed to run all the tests (if they were all scheduled in parallel)

In addition to feature work, we've fixed our readthedocs setup and added a changelog there - ducktape.readthedocs.io. We're also working on moving our planned work from internal confluent jira to the github issues/discussions, to be more transparent with the community and to give the community the way to ask for features or bugfixes.

[imcdo]

Thanks stan for writing all this up!