Merged
Conversation
- Automated benchmarking across multiple Foundry versions using hyperfine - Supports stable, nightly, and specific version tags (e.g., v1.0.0) - Benchmarks 5 major Foundry projects: account, v4-core, solady, morpho-blue, spark-psm - Tests forge test, forge build (no cache), and forge build (with cache) - Generates comparison tables in markdown format - Uses foundryup for version management - Exports JSON data for detailed analysis 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fix relative path issue causing JSON files to fail creation - Convert benchmark directories to absolute paths using SCRIPT_DIR - Improve markdown table formatting with proper column names and alignment - Use unified table generation with string concatenation for better formatting - Increase benchmark runs from 3 to 5 for more reliable results - Use --prepare instead of --cleanup for better cache management - Remove stderr suppression to catch hyperfine errors - Update table headers to show units (seconds) for clarity 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
3 tasks
0xrusowsky
reviewed
Jun 22, 2025
Contributor
0xrusowsky
left a comment
0xrusowsky
left a comment
There was a problem hiding this comment.
i think that the only "must" is support for env vars, the rest are nice-to-have features
- run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs
- Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
* feat: criterion benches * - setup benchmark repos in parallel - run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs * feat: shell script to run benches * feat: ci workflow, fix script * update readme * feat: enhance benchmarking suite with version flexibility - Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * latest bench * rm notes * remove shell based bench suite --------- Co-authored-by: Claude <noreply@anthropic.com>
* main.rs * forge version is controlled by the bin * parses criterion json to collect results - writes to LATEST.md
Contributor
📊 Foundry Benchmark Results📈 View all benchmark resultsFoundry Benchmark ResultsGenerated at: 2025-07-11 05:48:17 UTC Date: 2025-07-11 05:33:07 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Test
Forge Fuzz Test
System Information
Date: 2025-07-11 05:35:35 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Build (With Cache)
Forge Build (No Cache)
System Information
Date: 2025-07-11 05:38:31 SummaryBenchmarked 2 Foundry versions across 1 repositories. Repositories TestedFoundry Versions
Forge Coverage
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
Contributor
📊 Foundry Benchmark Results📈 View all benchmark resultsFoundry Benchmark ResultsGenerated at: 2025-07-11 05:48:40 UTC Date: 2025-07-11 05:47:53 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Test
Forge Fuzz Test
System Information
Date: 2025-07-11 05:41:02 SummaryBenchmarked 2 Foundry versions across 2 repositories. Repositories TestedFoundry Versions
Forge Build (No Cache)
Forge Build (With Cache)
System Information
Date: 2025-07-11 05:43:54 SummaryBenchmarked 2 Foundry versions across 1 repositories. Repositories TestedFoundry Versions
Forge Coverage
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
Contributor
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 08:33:33 UTC Forge TestRepositories TestedFoundry Versions
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <github-actions@github.com>
Contributor
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 10:28:32 UTC Forge TestRepositories TestedFoundry Versions
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <github-actions@github.com>
Contributor
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 10:49:57 UTC Forge TestRepositories TestedFoundry Versions
Forge Fuzz Test
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <github-actions@github.com>
Contributor
📊 Foundry Benchmark ResultsGenerated at: 2025-07-11 11:48:21 UTC Forge TestRepositories TestedFoundry Versions
Forge Fuzz Test
📈 View all benchmark resultsForge BuildRepositories TestedFoundry Versions
No Cache
With Cache
Forge CoverageRepositories TestedFoundry Versions
System Information
🤖 This comment was automatically generated by the Foundry Benchmarks workflow. To run benchmarks manually: Go to Actions → "Run workflow" |
github-actions
bot
dismissed stale reviews from mattsse and grandizzy
via
81cef23
grandizzy
approved these changes
Jul 11, 2025
0xrusowsky
approved these changes
Jul 11, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Closes #10548
Aggregated benchmark suite for testing performance of various commands such as test, build, coverage across multiple foundry versions and repositories.
Solution
/benchesbench/forge_test.rsBenchmarkProjectis the utility type to clone various repos and run forge commands on themfoundry-benchbinary which is a CLI that has flags to specify--versions,--repos,--benchmarks.BenchmarkResultstype aggregates the criterion results from thetarget/criterion/*cache.benchmarks.ymlworkflow that enables running benchmarks manuallyCurrently included
To be addressed in a followup
dynamic_test_linkingPR Checklist