diff --git a/README.md b/README.md index eb36f90..47a4a48 100644 --- a/README.md +++ b/README.md @@ -137,21 +137,46 @@ FlakeFighter has the following arguments. ### Enabling/Disabling the Plugin -By default, pytest-flakefighters runs whenever it is installed. To disable it for a specific test run, use: +To enable the plugin, run pytest with the `--flakefighters` argument ```bash -pytest --no-flakefighters +pytest --flakefighters ``` -This is useful when you have the plugin installed but want to run quick tests without flaky test detection. - You can also configure this in your `pyproject.toml`: ```toml [tool.pytest.ini_options] -addopts = "--no-flakefighters" +addopts = "--flakefighters" +``` + +### Configuration +By default, the plugin will only use the DeFlaker algorithm to classify flaky tests. +If you would like to use other algorithms as well (or instead), you need to configure these. +This can be done by adding appropriate fields in your pyproject.toml or pytest.ini file. +For example, you could add the following to your pyproject.toml. + +``` +[tool.pytest.ini_options.pytest_flakefighters.flakefighters.deflaker.DeFlaker] +run_live=true # run the classifier immediately after each test + +[tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.TracebackMatching] +run_live=false # run the classifier at the end of the test suite + +[tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.CosineSimilarity] +run_live=false # run the classifier at the end of the test suite +threshold=0.8 # Cosine similarity >= 0.8 is classed as a match + +[tool.pytest.ini_options.pytest_flakefighters.flakefighters.coverage_independence.CoverageIndependence] +run_live=false # run the classifier at the end of the test suite +threshold=0.1 # Distance <= 0.1 is classed as "similar" +metric=hamming # Use Hamming distance +linkage_method=complete # Use complete linkage for clustering ``` +> [!NOTE] +> The above configuration is just an example meant to demonstrate the various parameters that can be supplied, and is not a recommendation or "default". +> You should choose the parameter values that are appropriate for your project, especially threshold values for CosineSimilarity and CoverageIndependence. ## Contributing diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst index d4b2873..b3b9801 100644 --- a/docs/source/configuration.rst +++ b/docs/source/configuration.rst @@ -3,16 +3,32 @@ Configuration The flakefighters plugin implements several cutting edge flaky test detection tools from the research community. Each one is individually configurable and can be run individually or with other flakefighters. +By default, the plugin will only use the DeFlaker algorithm to classify flaky tests. You can control which flakefighters to run and provide additional configuration options from your :code:`pyproject.toml` file by including sections of the following form for each flakefighter you want to run. Here, :code:`` is the class of the flakefighter you wish to configure as if you were going to import it into a source code file. -.. code-block:: ini +.. code-block:: toml - [tool.pytest.ini_options.pytest_flakefighters.] - run_live=[true/false] - option1=value1 - option2=value2 - ... + [tool.pytest.ini_options.pytest_flakefighters.flakefighters.deflaker.DeFlaker] + run_live=true # run the classifier immediately after each test + + [tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.TracebackMatching] + run_live=false # run the classifier at the end of the test suite + + [tool.pytest.ini_options.pytest_flakefighters.flakefighters.traceback_matching.CosineSimilarity] + run_live=false # run the classifier at the end of the test suite + threshold=0.8 # Cosine similarity >= 0.8 is classed as a match + + [tool.pytest.ini_options.pytest_flakefighters.flakefighters.coverage_independence.CoverageIndependence] + run_live=false # run the classifier at the end of the test suite + threshold=0.1 # Distance <= 0.1 is classed as "similar" + metric=hamming # Use Hamming distance + linkage_method=complete # Use complete linkage for clustering + +.. note:: + The above configuration is just an example meant to demonstrate the various parameters that can be supplied, and is not a recommendation or "default". + + You should choose the parameter values that are appropriate for your project, especially threshold values for **CosineSimilarity** and **CoverageIndependence**. Every flakefighter has a :code:`run_live` option, which can be set to :code:`true` to classify each test execution as flaky immediately after it is run, or :code:`false` to clasify all tests at once at the end, although individual flakefighters may only support one of these. Individual flakefighters have their own configurable options. diff --git a/src/pytest_flakefighters/flakefighters/traceback_matching.py b/src/pytest_flakefighters/flakefighters/traceback_matching.py index fec381b..0a520cd 100644 --- a/src/pytest_flakefighters/flakefighters/traceback_matching.py +++ b/src/pytest_flakefighters/flakefighters/traceback_matching.py @@ -120,6 +120,10 @@ class CosineSimilarity(TracebackMatching): Test executions are classified as flaky if the stack trace is sufficiently similar to a previous flaky execution. :ivar run_live: Run detection "live" after each test. Otherwise run as a postprocessing step after the test suite. + :ivar previous_runs: List of previous FlakeFighters runs. + :ivar root: The root directory of the code repository. + :ivar threshold: The minimum distance to consider as "similar", expressed as a proportion 0 <= threshold < 1 where 0 + represents no difference and 1 represents complete difference. """ def __init__(self, run_live: bool, previous_runs: list[Run], root: str = ".", threshold: float = 1):