Skip to content

Conversation

@ctoyan
Copy link
Member

@ctoyan ctoyan commented Oct 16, 2025

Implements configurable mutation operator modes (light, medium, heavy) based on operator effectiveness analysis, which is at the end of the description below, (22,597 mutants across Aptos' Move framework). This allows users to trade off speed vs thoroughness when running mutation tests. It also allows to add different operators that we support, per operator basis. Changes applied to the move-mutator and move-mutation-test CLI tools.

Motivation

Running full mutation testing with all operators can be time-consuming.
This PR enables faster feedback loops by allowing users to run only the most effective operators or custom ones that they want to test for.

Changes

CLI Additions

Both move-mutator and move-mutation-test now support:

--mode <light|medium|heavy> - Predefined operator sets

  • light: binary_operator_swap, break_continue_replacement, delete_statement
  • medium: light + literal_replacement
  • medium-only - only the operator added in medium (not including light)
  • heavy (default): all
  • heavy-only - only the operators added in heavy (not including light/medium)

--operators <op1,op2,...> - Custom operator selection

  • Comma-separated list of specific operators
  • Validates operator names and aborts on invalid input
  • Mutually exclusive with --mode

Usage Examples

# Light mode - fastest execution (3 operators)
move-mutation-test run --mode light

# Custom selection
move-mutation-test run --operators unary_operator_replacement,delete_statement

Operator effectiveness analysis

The data below was calculated by running the tool on projects in Aptos' Move Framework.

The "Kill Rate" column shows the percentage of mutants killed by tests. While a high kill rate indicates good test coverage, operators with lower kill rates (more surviving mutants) are often more effective at detecting test gaps, as surviving mutants reveal untested code paths.

Analysis is on the following projects:
----------------------------------------
aptos-core/aptos-move/framework/aptos-experimental
aptos-core/aptos-move/framework/aptos-framework
aptos-core/aptos-move/framework/aptos-stdlib
aptos-core/aptos-move/framework/aptos-token
aptos-core/aptos-move/framework/aptos-token-objects
aptos-core/aptos-move/framework/move-stdlib

Summary:
----------------------------------------
Total mutants tested: 22597
Total mutants killed: 18535
Overall effectiveness: 82.02%


Operator Effectiveness Rankings:
----------------------------------------
╭──────┬─────────────────────────────┬────────┬────────┬───────────────┬───────────╮
│ Rank │ Operator                    │ Tested │ Killed │ Kill Rate     │ Survived  │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #1   │ unary_operator_replacement  │ 219    │ 219    │ 100.00%       │ 0/219     │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #2   │ delete_statement            │ 909    │ 895    │ 98.46%        │ 14/909    │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #3   │ break_continue_replacement  │ 26     │ 23     │ 88.46%        │ 3/26      │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #4   │ binary_operator_replacement │ 7081   │ 6207   │ 87.66%        │ 874/7081  │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #5   │ if_else_replacement         │ 5310   │ 4579   │ 86.23%        │ 731/5310  │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #6   │ literal_replacement         │ 8781   │ 6498   │ 74.00%        │ 2283/8781 │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #7   │ binary_operator_swap        │ 271    │ 114    │ 42.07%        │ 157/271   │
╰──────┴─────────────────────────────┴────────┴────────┴───────────────┴───────────╯

@ctoyan ctoyan self-assigned this Oct 16, 2025
@ctoyan ctoyan requested a review from joske October 16, 2025 19:08
joske
joske previously approved these changes Oct 17, 2025
@ctoyan
Copy link
Member Author

ctoyan commented Oct 17, 2025

The --operators is something outside of the scope, but I think it's a nice addition, because it gives users more flexibility to test different operator types.

And for the --mode, we figured out that this operation combination for each mode is a good balance between speed and testing the mutants that survive the most, starting from light to heavy.

joske
joske previously approved these changes Oct 20, 2025
joske
joske previously approved these changes Oct 23, 2025
@ctoyan ctoyan merged commit 992572e into main Oct 27, 2025
4 checks passed
@ctoyan ctoyan deleted the dev-m4 branch October 27, 2025 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants