Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions move-mutation-test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,5 +161,26 @@ If the user wants to mutate only the `sum` function in the `Sum` module, the use
RUST_LOG=info ./target/release/move-mutation-test run --package-dir move-mutator/tests/move-assets/simple --output report.txt --move-2 --mutate-functions sum --mutate-modules Sum
./target/release/move-mutation-test display-report coverage --path-to-report report.txt --modules Sum
```
------------------------------------------------------------------------------------------------------------
To speed up mutation testing by using only the [most effective operators](../move-mutator/doc/design.md#operator-effectiveness-analysis), use the `--mode` option:
```bash
# Light mode - fastest, uses only top 3 most effective operators
RUST_LOG=info ./target/release/move-mutation-test run --package-dir move-mutator/tests/move-assets/simple --output report.txt --mode light

# Medium mode - balanced, uses top 5 most effective operators
RUST_LOG=info ./target/release/move-mutation-test run --package-dir move-mutator/tests/move-assets/simple --output report.txt --mode medium

# Heavy mode - default, uses all operators
RUST_LOG=info ./target/release/move-mutation-test run --package-dir move-mutator/tests/move-assets/simple --output report.txt --mode heavy
```
------------------------------------------------------------------------------------------------------------
For fine-grained control over which operators to apply, use the `--operators` option with a comma-separated list:
```bash
RUST_LOG=info ./target/release/move-mutation-test run --package-dir move-mutator/tests/move-assets/simple --output report.txt --operators delete_statement,binary_operator_replacement,if_else_replacement
```

Available operators: `unary_operator_replacement`, `delete_statement`, `break_continue_replacement`, `binary_operator_replacement`, `if_else_replacement`, `literal_replacement`, `binary_operator_swap`.

**Note:** The `--mode` and `--operators` options are mutually exclusive.

[nextest]: https://github.com/nextest-rs/nextest
29 changes: 28 additions & 1 deletion move-mutation-test/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use aptos::common::types::MovePackageOptions;
use aptos_framework::extended_checks;
use clap::Parser;
use move_model::metadata::{CompilerVersion, LanguageVersion};
use move_mutator::cli::{FunctionFilter, ModuleFilter};
use move_mutator::cli::{FunctionFilter, ModuleFilter, OperatorModeArg};
use move_package::CompilerConfig;
use std::path::PathBuf;

Expand Down Expand Up @@ -42,6 +42,31 @@ pub struct CLIOptions {
/// Remove averagely given percentage of mutants. See the doc for more details.
#[clap(long, conflicts_with = "use_generated_mutants")]
pub downsampling_ratio_percentage: Option<usize>,

/// Mutation operator mode: light (fastest), medium (balanced), or heavy (full coverage, default).
///
/// - light: unary_operator_replacement, delete_statement, break_continue_replacement
/// - medium: light + binary_operator_replacement, if_else_replacement
/// - heavy (default): medium + literal_replacement, binary_operator_swap
#[clap(
long,
value_enum,
conflicts_with = "operators",
conflicts_with = "use_generated_mutants"
)]
pub mode: Option<OperatorModeArg>,

/// Custom operator selection to run mutations on (comma-separated).
///
/// Available operators: unary_operator_replacement, delete_statement, break_continue_replacement, binary_operator_replacement, if_else_replacement,w literal_replacement, binary_operator_swap
#[clap(
long,
value_parser,
value_delimiter = ',',
conflicts_with = "mode",
conflicts_with = "use_generated_mutants"
)]
pub operators: Option<Vec<String>>,
}

/// This function creates a mutator CLI options from the given mutation-test options.
Expand All @@ -57,6 +82,8 @@ pub fn create_mutator_options(
apply_coverage,
// To run tests, compilation must succeed
verify_mutants: true,
mode: options.mode.clone(),
operators: options.operators.clone(),
..Default::default()
}
}
Expand Down
31 changes: 31 additions & 0 deletions move-mutator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,34 @@ directory. They can be used to check the mutator tool as well.
To check possible options, use the `--help` option.

[nextest]: https://github.com/nextest-rs/nextest

### Operator modes

The mutator tool supports different operator modes to control which mutation operators are applied. This can significantly reduce mutation testing time by focusing on the most effective operators.

Use the `--mode` option to select a predefined operator set:
```bash
# Light mode: fastest, uses only the top 3 most effective operators
./target/release/move-mutator --package-dir move-mutator/tests/move-assets/simple/ --mode light

# Medium mode: balanced, uses the top 5 most effective operators
./target/release/move-mutator --package-dir move-mutator/tests/move-assets/simple/ --mode medium

# Heavy mode (default): uses all 7 available operators
./target/release/move-mutator --package-dir move-mutator/tests/move-assets/simple/ --mode heavy
```

The operator modes are based on [effectiveness analysis](doc/design.md#operator-effectiveness-analysis) where:
- **light**: `unary_operator_replacement`, `delete_statement`, `break_continue_replacement`
- **medium**: light + `binary_operator_replacement`, `if_else_replacement`
- **heavy**: medium + `literal_replacement`, `binary_operator_swap`

For fine-grained control, use the `--operators` option to specify exactly which operators to apply:
```bash
# Apply only specific operators
./target/release/move-mutator --package-dir move-mutator/tests/move-assets/simple/ --operators delete_statement,binary_operator_replacement,if_else_replacement
```

Available operators: `unary_operator_replacement`, `delete_statement`, `break_continue_replacement`, `binary_operator_replacement`, `if_else_replacement`, `literal_replacement`, `binary_operator_swap`.

**Note:** The `--mode` and `--operators` options are mutually exclusive.
53 changes: 53 additions & 0 deletions move-mutator/doc/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,59 @@ inefficient. Once mutation places are identified, mutants are generated in
reversed order (based on localization) to avoid that. Tools are ready to be
extended to support the operator mixing, if needed.

### Operator filtering

The Move mutator tool supports operator filtering to control which mutation
operators are applied during the mutation process. This feature allows users to
focus on specific operators or use predefined modes based on [operator
effectiveness](#operator-effectiveness-analysis).

Three predefined modes are available:
- **Light mode**: Uses the top 3 most effective operators (approximately 95% faster than heavy mode)
- **Medium mode**: Uses the top 5 most effective operators (approximately 40% faster than heavy mode)
- **Heavy mode** (default): Uses all 7 available operators

Users can also specify custom operator sets using the `--operators` CLI option,
providing a comma-separated list of operator names. This allows for fine-grained
control over which operators are applied.

Operator filtering is performed during AST traversal in the `mutate.rs` module.
When a potential mutation site is found, the tool checks if the corresponding
operator is enabled in the current mode before creating the mutant. This approach
prevents unnecessary mutant generation, making the process more efficient.

#### Operator effectiveness analysis

The effectiveness rankings were calculated by running the tool on the largest
projects in [Aptos' Move Framework](https://github.com/aptos-labs/aptos-core/tree/main/aptos-move/framework), testing 22,597 mutants with an overall kill
rate of 82.02%. Operators are ranked by their effectiveness (percentage of
mutants killed by tests):

```
╭──────┬─────────────────────────────┬────────┬────────┬───────────────┬───────────╮
│ Rank │ Operator │ Tested │ Killed │ Effectiveness │ Kill Rate │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #1 │ unary_operator_replacement │ 219 │ 219 │ 100.00% │ 219/219 │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #2 │ delete_statement │ 909 │ 895 │ 98.46% │ 895/909 │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #3 │ break_continue_replacement │ 26 │ 23 │ 88.46% │ 23/26 │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #4 │ binary_operator_replacement │ 7081 │ 6207 │ 87.66% │ 6207/7081 │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #5 │ if_else_replacement │ 5310 │ 4579 │ 86.23% │ 4579/5310 │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #6 │ literal_replacement │ 8781 │ 6498 │ 74.00% │ 6498/8781 │
├──────┼─────────────────────────────┼────────┼────────┼───────────────┼───────────┤
│ #7 │ binary_operator_swap │ 271 │ 114 │ 42.07% │ 114/271 │
╰──────┴─────────────────────────────┴────────┴────────┴───────────────┴───────────╯
```

The predefined operator modes are based on this analysis:
- **Light mode** includes operators #1-3 (effectiveness ≥88%)
- **Medium mode** includes operators #1-5 (effectiveness ≥86%)
- **Heavy mode** includes all operators #1-7

The Move mutator tool implements the following mutation operators.

### Binary operator replacement
Expand Down
26 changes: 25 additions & 1 deletion move-mutator/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,19 @@
// Copyright © Aptos Foundation
// SPDX-License-Identifier: Apache-2.0

use clap::Parser;
use clap::{Parser, ValueEnum};
use std::{path::PathBuf, str::FromStr};

pub const DEFAULT_OUTPUT_DIR: &str = "mutants_output";

/// Mutation operator mode
#[derive(Debug, Clone, Copy, PartialEq, Eq, ValueEnum)]
pub enum OperatorModeArg {
Light,
Medium,
Heavy,
}

/// Command line options for mutator
#[derive(Parser, Debug, Clone)]
pub struct CLIOptions {
Expand Down Expand Up @@ -41,6 +49,20 @@ pub struct CLIOptions {
/// Use the unit test coverage report to generate mutants for source code with unit test coverage.
#[clap(long = "coverage", conflicts_with = "move_sources")]
pub apply_coverage: bool,

/// Mutation operator mode: light (fastest), medium (balanced), or heavy (full coverage, default).
///
/// - light: unary_operator_replacement, delete_statement, break_continue_replacement
/// - medium: light + binary_operator_replacement, if_else_replacement
/// - heavy (default): medium + literal_replacement, binary_operator_swap
#[clap(long, value_enum, conflicts_with = "operators")]
pub mode: Option<OperatorModeArg>,

/// Custom operator selection to run mutations on (comma-separated).
///
/// Available operators: unary_operator_replacement, delete_statement, break_continue_replacement, binary_operator_replacement, if_else_replacement,w literal_replacement, binary_operator_swap
#[clap(long, value_parser, value_delimiter = ',', conflicts_with = "mode")]
pub operators: Option<Vec<String>>,
}

/// Checker for conflicts with CLI arguments.
Expand Down Expand Up @@ -82,6 +104,8 @@ impl Default for CLIOptions {
no_overwrite: false,
apply_coverage: false,
downsampling_ratio_percentage: None,
mode: None,
operators: None,
}
}
}
Expand Down
55 changes: 50 additions & 5 deletions move-mutator/src/configuration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,74 @@
// Copyright © Aptos Foundation
// SPDX-License-Identifier: Apache-2.0

use crate::{cli::CLIOptions, coverage::Coverage};
use crate::{
cli::{CLIOptions, OperatorModeArg},
coverage::Coverage,
operator_filter::OperatorMode,
};
use std::path::PathBuf;

/// Mutator configuration for the Move project.
#[derive(Debug, Default)]
#[derive(Debug)]
pub struct Configuration {
/// Main project options. It's the same as the CLI options.
pub project: CLIOptions,
/// Path to the project.
pub project_path: Option<PathBuf>,
/// Coverage report where the optional unit test coverage data is stored.
pub(crate) coverage: Coverage,
/// Operator filter that determines which mutation operators are enabled.
pub operator_mode: OperatorMode,
}

impl Configuration {
/// Creates a new configuration using command line options.
#[must_use]
pub fn new(project: CLIOptions, project_path: Option<PathBuf>) -> Self {
Self {
pub fn new(project: CLIOptions, project_path: Option<PathBuf>) -> anyhow::Result<Self> {
// Parse and validate the operator mode from CLI options
let operator_mode = Self::parse_operator_mode(&project)?;

Ok(Self {
project,
project_path,
// Coverage is disabled by default.
coverage: Coverage::default(),
operator_mode,
})
}

fn parse_operator_mode(project: &CLIOptions) -> anyhow::Result<OperatorMode> {
match (&project.mode, &project.operators) {
// --operators specified
(None, Some(operators)) => {
let parsed_ops = OperatorMode::parse_operators(operators)?;
Ok(OperatorMode::Custom(parsed_ops))
},
// --mode specified
(Some(mode_arg), None) => {
let mode = match mode_arg {
OperatorModeArg::Light => OperatorMode::Light,
OperatorModeArg::Medium => OperatorMode::Medium,
OperatorModeArg::Heavy => OperatorMode::Heavy,
};
Ok(mode)
},
// neither specified
(None, None) => Ok(OperatorMode::default()),
// both specified - this should be prevented by clap conflicts
(Some(_), Some(_)) => {
unreachable!("Both --mode and --operators specified")
},
}
}
}

impl Default for Configuration {
fn default() -> Self {
Self {
project: CLIOptions::default(),
project_path: None,
coverage: Coverage::default(),
operator_mode: OperatorMode::default(),
}
}
}
12 changes: 11 additions & 1 deletion move-mutator/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ pub mod configuration;
pub(crate) mod coverage;
mod mutant;
mod operator;
pub mod operator_filter;
mod operators;
mod output;
pub mod report;
Expand Down Expand Up @@ -74,10 +75,19 @@ pub fn run_move_mutator(
};

let mut mutator_configuration =
Configuration::new(options, Some(original_package_path.to_owned()));
Configuration::new(options, Some(original_package_path.to_owned()))?;

trace!("Mutator configuration: {mutator_configuration:?}");

let enabled_operators = mutator_configuration.operator_mode.get_operators();
println!(
"Operator types being mutated ({}):",
enabled_operators.len()
);
for (i, op) in enabled_operators.iter().enumerate() {
println!(" {}. {}", i + 1, op);
}

let package_path = mutator_configuration
.project_path
.clone()
Expand Down
Loading