Use mutation testing to find the bugs your tests don't catch

[carlospolop]

🤖 Automated Content Update

This PR was automatically generated by the HackTricks News Bot based on a technical blog post.

📝 Source Information

• Blog URL: https://blog.trailofbits.com/2025/09/18/use-mutation-testing-to-find-the-bugs-your-tests-dont-catch/
• Blog Title: Use mutation testing to find the bugs your tests don't catch
• Suggested Section: Blockchain & Crypto -> Smart Contract Security -> Testing & Analysis -> Mutation Testing with Slither (slither-mutate)

🎯 Content Summary

Thesis. Line/branch coverage proves code was executed, not that behavior is correct. The post demonstrates how mutation testing (via Slither’s slither-mutate) exposes gaps that high coverage misses, and shows how it led Trail of Bits to a high‑severity bug in the Arkis DeFi protocol.

How tests improve security.
• Unit tests: assert behavior of individual functions and edge inputs.
• Integration tests: assert end‑to‑end user/admin flo...

🔧 Technical Details

Mutation testing to reveal test-suite blind spots. Systematically mutate Solidity code (operators, constants, conditions, comment/revert/type changes) and run the existing tests. Any surviving mutant marks a missing assertion. Automate with slither-mutate ./src/contracts --test-cmd="forge test", list mutators via --list-mutators, and prioritize mutants to cut runtime. Inspect survivors in ./mutation_campaign, then add assertions/invariants that would fail under the mutated behavior.

Assert state, not just return values. Strengthen tests with equality‑boundary checks (e.g., threshold ==), and verify post‑conditions: balances, supply, authorizations, and emitted events. This kills mutants that alter thresholds, skip lines, or negate conditions while still returning plausible values.

Validate user-controlled parameters against on-chain effects. When consuming user‑provided structs/fields (e.g...

🤖 Agent Actions

I analyzed the Trail of Bits post and confirmed it is highly technical and directly relevant to Smart Contract Security testing. There was no existing HackTricks content covering mutation testing for Solidity or Slither’s slither-mutate. I created a new page and linked it from relevant Blockchain pages.

What I added

• New page: Mutation Testing for Solidity with Slither (slither-mutate)
  • Path: src/blockchain/smart-contract-security/testing-and-analysis/mutation-testing-with-slither.md
  • Content:
    • Why code coverage is insufficient, with a Solidity threshold example.
    • How mutation testing “tests your tests” and measures effectiveness (% mutants killed).
    • Practical usage with slither-mutate including:
      • Example command for Foundry: slither-mutate ./src/contracts --test-cmd="forge test" &> >(tee mutation.results)
      • How to discover options and mutators: --help, --list-mutators
      • Output explanation and artifact paths (./mutation_campaign).
    • Common mutators (operator/assignment/constant replacement, condition negation, comment/revert/type swaps).
    • Performance strategies: prioritizing mutants, scoping, parallelization, fail-fast.
    • Triage workflow: assert state not just return values; boundary checks; invariants; replace unrealistic mocks; iterate until all mutants are killed or justified.
    • Case study summary: Arkis audit survivor indicating missing post-state assertions (user-controlled _cmd.value), impact on solvency.
    • Severity guidance: treat survivors affecting value transfer/accounting/authz as high risk.
    • References to the blog, Arkis audit PDF, and Slither repo.
  • Proper HackTricks formatting (banners, headings, code blocks, References section).

Where I linked it

• src/blockchain/blockchain-and-crypto-currencies/README.md
  • Added “Smart Contract Security” section with an internal reference to the new mutation testing page.
• src/crypto-and-stego/blockchain-and-crypto-currencies.md
  • Added the same “Smart Contract Security” section referencing the new page (correct relative path).

Why this location

• No existing Smart Contract Security or Solidity testing content was present. Creating a focused, technique-driven page under Blockchain → Smart Contract Security → Testing and Analysis aligns with HackTricks’ structure and the blog’s content.

References included on the new page

• Trail of Bits blog post
• Arkis DeFi audit PDF (Appendix C)
• Slither GitHub repository

No other duplicate content existed, so this addition fills a clear gap without redundancy.

---

This PR was automatically created by the HackTricks Feed Bot. Please review the changes carefully before merging.

🗑️ File Cleanup

• searchindex.js removed: This auto-generated file has been removed to prevent conflicts.

[carlospolop]

🔗 Additional Context

Original Blog Post: https://blog.trailofbits.com/2025/09/18/use-mutation-testing-to-find-the-bugs-your-tests-dont-catch/

Content Categories: Based on the analysis, this content was categorized under "Blockchain & Crypto -> Smart Contract Security -> Testing & Analysis -> Mutation Testing with Slither (slither-mutate)".

Repository Maintenance:

• MD Files Formatting: 877 files processed

Review Notes:

• This content was automatically processed and may require human review for accuracy
• Check that the placement within the repository structure is appropriate
• Verify that all technical details are correct and up-to-date
• All .md files have been checked for proper formatting (headers, includes, etc.)

Bot Version: HackTricks News Bot v1.0