Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 30 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,21 +137,46 @@ FlakeFighter has the following arguments.

### Enabling/Disabling the Plugin

By default, pytest-flakefighters runs whenever it is installed. To disable it for a specific test run, use:
To enable the plugin, run pytest with the `--flakefighters` argument

```bash
pytest --no-flakefighters
pytest --flakefighters
```

This is useful when you have the plugin installed but want to run quick tests without flaky test detection.

You can also configure this in your `pyproject.toml`:

```toml
[tool.pytest.ini_options]
addopts = "--no-flakefighters"
addopts = "--flakefighters"
```

### Configuration
By default, the plugin will only use the DeFlaker algorithm to classify flaky tests.
If you would like to use other algorithms as well (or instead), you need to configure these.
This can be done by adding appropriate fields in your pyproject.toml or pytest.ini file.
For example, you could add the following to your pyproject.toml.

```
[tool.pytest.ini_options.pytest_flakefighters.flakefighters.deflaker.DeFlaker]
run_live=true # run the classifier immediately after each test

[tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.TracebackMatching]
run_live=false # run the classifier at the end of the test suite

[tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.CosineSimilarity]
run_live=false # run the classifier at the end of the test suite
threshold=0.8 # Cosine similarity >= 0.8 is classed as a match

[tool.pytest.ini_options.pytest_flakefighters.flakefighters.coverage_independence.CoverageIndependence]
run_live=false # run the classifier at the end of the test suite
threshold=0.1 # Distance <= 0.1 is classed as "similar"
metric=hamming # Use Hamming distance
linkage_method=complete # Use complete linkage for clustering
```

> [!NOTE]
> The above configuration is just an example meant to demonstrate the various parameters that can be supplied, and is not a recommendation or "default".
> You should choose the parameter values that are appropriate for your project, especially threshold values for CosineSimilarity and CoverageIndependence.

## Contributing

Expand Down
28 changes: 22 additions & 6 deletions docs/source/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,32 @@ Configuration

The flakefighters plugin implements several cutting edge flaky test detection tools from the research community.
Each one is individually configurable and can be run individually or with other flakefighters.
By default, the plugin will only use the DeFlaker algorithm to classify flaky tests.
You can control which flakefighters to run and provide additional configuration options from your :code:`pyproject.toml` file by including sections of the following form for each flakefighter you want to run.
Here, :code:`<FlakeFighterClass>` is the class of the flakefighter you wish to configure as if you were going to import it into a source code file.

.. code-block:: ini
.. code-block:: toml

[tool.pytest.ini_options.pytest_flakefighters.<FlakeFighterClass>]
run_live=[true/false]
option1=value1
option2=value2
...
[tool.pytest.ini_options.pytest_flakefighters.flakefighters.deflaker.DeFlaker]
run_live=true # run the classifier immediately after each test

[tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.TracebackMatching]
run_live=false # run the classifier at the end of the test suite

[tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.CosineSimilarity]
run_live=false # run the classifier at the end of the test suite
threshold=0.8 # Cosine similarity >= 0.8 is classed as a match

[tool.pytest.ini_options.pytest_flakefighters.flakefighters.coverage_independence.CoverageIndependence]
run_live=false # run the classifier at the end of the test suite
threshold=0.1 # Distance <= 0.1 is classed as "similar"
metric=hamming # Use Hamming distance
linkage_method=complete # Use complete linkage for clustering

.. note::
The above configuration is just an example meant to demonstrate the various parameters that can be supplied, and is not a recommendation or "default".

You should choose the parameter values that are appropriate for your project, especially threshold values for **CosineSimilarity** and **CoverageIndependence**.

Every flakefighter has a :code:`run_live` option, which can be set to :code:`true` to classify each test execution as flaky immediately after it is run, or :code:`false` to clasify all tests at once at the end, although individual flakefighters may only support one of these.
Individual flakefighters have their own configurable options.
Expand Down
4 changes: 4 additions & 0 deletions src/pytest_flakefighters/flakefighters/traceback_matching.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,10 @@ class CosineSimilarity(TracebackMatching):
Test executions are classified as flaky if the stack trace is sufficiently similar to a previous flaky execution.

:ivar run_live: Run detection "live" after each test. Otherwise run as a postprocessing step after the test suite.
:ivar previous_runs: List of previous FlakeFighters runs.
:ivar root: The root directory of the code repository.
:ivar threshold: The minimum distance to consider as "similar", expressed as a proportion 0 <= threshold < 1 where 0
represents no difference and 1 represents complete difference.
"""

def __init__(self, run_live: bool, previous_runs: list[Run], root: str = ".", threshold: float = 1):
Expand Down