Dataset for evaluation

[jmafoster1]

Mine existing python projects for CI builds that failed, were restarted, and subsequently passed. See if we can find some with issues/bug reports. Look at the original DeFlaker paper. Comments about flaky tests. (We'd have to be able to reproduce the flakiness).

Previous work: The same state but run many times.
Our work: Changes to a project - two project states.

Home assistant project?
https://github.com/home-assistant/core

[jmafoster1]

Where do real failures come from?

• Find commits with genuine bugs in (crawl the repo)
• Could seed faults, but not as convincing

Want a relatively balanced number of each type of test. Likely to find vastly more genuine failures than flaky failures.

Is there value in doing a broad survey of projects to show that it installs and runs on a wide variety of projects with different coverage, test suite size, coverage, histories, etc.
How to find suitable projects?

• https://github.com/ossf/criticality_score
• https://commondatastorage.googleapis.com/ossf-criticality-score/index.html (Download the CSV, filter out the python ones, spend a few minutes trying to set each one up)
• Look at previous flaky test studies that have used python subjects and just use those