Fix `tests bench [--opt=ALL]` so it runs opt and non-opt benchmarks #657

hanno-becker · 2025-01-13T19:33:16Z

Fixes tests bench --opt ALL -c NO not executing no_opt benchmarks #652

Previously, tests bench and tests bench --opt=ALL would only run the opt benchmarks. This was because of a bug in the helper do_opt_all() in the tests script. This commit fixes this bug, thereby restoring the desired behaviour.

Previously, `tests bench` and `tests bench --opt=ALL` would only run the opt benchmarks. This was because of a bug in the helper `do_opt_all()` in the `tests` script. This commit fixes this bug, thereby restoring the desired behaviour. Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>

mkannwischer

LGTM - it does what it should now on my machine.
Re-running benchmarking CI just to be sure everything is fine.

oqs-bot

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`29050` cycles	`28987` cycles	`1.00`
`ML-KEM-512 encaps`	`35408` cycles	`35400` cycles	`1.00`
`ML-KEM-512 decaps`	`45884` cycles	`45899` cycles	`1.00`
`ML-KEM-768 keypair`	`49369` cycles	`49365` cycles	`1.00`
`ML-KEM-768 encaps`	`55573` cycles	`55563` cycles	`1.00`
`ML-KEM-768 decaps`	`70337` cycles	`70316` cycles	`1.00`
`ML-KEM-1024 keypair`	`72077` cycles	`71987` cycles	`1.00`
`ML-KEM-1024 encaps`	`80874` cycles	`80740` cycles	`1.00`
`ML-KEM-1024 decaps`	`100731` cycles	`100610` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`13536` cycles	`13509` cycles	`1.00`
`ML-KEM-512 encaps`	`17394` cycles	`17245` cycles	`1.01`
`ML-KEM-512 decaps`	`22929` cycles	`22768` cycles	`1.01`
`ML-KEM-768 keypair`	`22558` cycles	`22550` cycles	`1.00`
`ML-KEM-768 encaps`	`24574` cycles	`24541` cycles	`1.00`
`ML-KEM-768 decaps`	`32567` cycles	`32551` cycles	`1.00`
`ML-KEM-1024 keypair`	`31382` cycles	`31368` cycles	`1.00`
`ML-KEM-1024 encaps`	`34905` cycles	`34912` cycles	`1.00`
`ML-KEM-1024 decaps`	`45834` cycles	`45828` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`18099` cycles	`18083` cycles	`1.00`
`ML-KEM-512 encaps`	`23129` cycles	`23010` cycles	`1.01`
`ML-KEM-512 decaps`	`30336` cycles	`30198` cycles	`1.00`
`ML-KEM-768 keypair`	`31100` cycles	`31092` cycles	`1.00`
`ML-KEM-768 encaps`	`33948` cycles	`33870` cycles	`1.00`
`ML-KEM-768 decaps`	`44504` cycles	`44542` cycles	`1.00`
`ML-KEM-1024 keypair`	`44769` cycles	`44702` cycles	`1.00`
`ML-KEM-1024 encaps`	`49965` cycles	`49922` cycles	`1.00`
`ML-KEM-1024 decaps`	`64363` cycles	`64367` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`20351` cycles	`20347` cycles	`1.00`
`ML-KEM-512 encaps`	`26967` cycles	`26950` cycles	`1.00`
`ML-KEM-512 decaps`	`35745` cycles	`35738` cycles	`1.00`
`ML-KEM-768 keypair`	`34891` cycles	`34909` cycles	`1.00`
`ML-KEM-768 encaps`	`38175` cycles	`38169` cycles	`1.00`
`ML-KEM-768 decaps`	`50955` cycles	`50936` cycles	`1.00`
`ML-KEM-1024 keypair`	`48016` cycles	`47937` cycles	`1.00`
`ML-KEM-1024 encaps`	`54192` cycles	`54085` cycles	`1.00`
`ML-KEM-1024 decaps`	`71689` cycles	`71587` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`14905` cycles	`14924` cycles	`1.00`
`ML-KEM-512 encaps`	`19664` cycles	`19651` cycles	`1.00`
`ML-KEM-512 decaps`	`26303` cycles	`26305` cycles	`1.00`
`ML-KEM-768 keypair`	`25621` cycles	`25579` cycles	`1.00`
`ML-KEM-768 encaps`	`28093` cycles	`28104` cycles	`1.00`
`ML-KEM-768 decaps`	`37831` cycles	`37808` cycles	`1.00`
`ML-KEM-1024 keypair`	`35676` cycles	`35738` cycles	`1.00`
`ML-KEM-1024 encaps`	`40974` cycles	`40964` cycles	`1.00`
`ML-KEM-1024 decaps`	`54504` cycles	`54429` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i) (no-opt)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`34986` cycles	`35025` cycles	`1.00`
`ML-KEM-512 encaps`	`46206` cycles	`46092` cycles	`1.00`
`ML-KEM-512 decaps`	`58765` cycles	`58751` cycles	`1.00`
`ML-KEM-768 keypair`	`58921` cycles	`58921` cycles	`1`
`ML-KEM-768 encaps`	`70870` cycles	`70868` cycles	`1.00`
`ML-KEM-768 decaps`	`88464` cycles	`88528` cycles	`1.00`
`ML-KEM-1024 keypair`	`86954` cycles	`87091` cycles	`1.00`
`ML-KEM-1024 encaps`	`104297` cycles	`104277` cycles	`1.00`
`ML-KEM-1024 decaps`	`126903` cycles	`127045` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`18966` cycles	`18960` cycles	`1.00`
`ML-KEM-512 encaps`	`23559` cycles	`23574` cycles	`1.00`
`ML-KEM-512 decaps`	`30691` cycles	`30661` cycles	`1.00`
`ML-KEM-768 keypair`	`32314` cycles	`32311` cycles	`1.00`
`ML-KEM-768 encaps`	`35887` cycles	`35894` cycles	`1.00`
`ML-KEM-768 decaps`	`46032` cycles	`46028` cycles	`1.00`
`ML-KEM-1024 keypair`	`46620` cycles	`46639` cycles	`1.00`
`ML-KEM-1024 encaps`	`52442` cycles	`52460` cycles	`1.00`
`ML-KEM-1024 decaps`	`66278` cycles	`66273` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`51704` cycles	`51715` cycles	`1.00`
`ML-KEM-512 encaps`	`58247` cycles	`58028` cycles	`1.00`
`ML-KEM-512 decaps`	`74525` cycles	`74871` cycles	`1.00`
`ML-KEM-768 keypair`	`87949` cycles	`87861` cycles	`1.00`
`ML-KEM-768 encaps`	`96703` cycles	`95951` cycles	`1.01`
`ML-KEM-768 decaps`	`119510` cycles	`119765` cycles	`1.00`
`ML-KEM-1024 keypair`	`131892` cycles	`131877` cycles	`1.00`
`ML-KEM-1024 encaps`	`144736` cycles	`145008` cycles	`1.00`
`ML-KEM-1024 decaps`	`175904` cycles	`175836` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a) (no-opt)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`52578` cycles	`52392` cycles	`1.00`
`ML-KEM-512 encaps`	`65013` cycles	`65292` cycles	`1.00`
`ML-KEM-512 decaps`	`88252` cycles	`88098` cycles	`1.00`
`ML-KEM-768 keypair`	`84552` cycles	`84994` cycles	`0.99`
`ML-KEM-768 encaps`	`102077` cycles	`101512` cycles	`1.01`
`ML-KEM-768 decaps`	`131298` cycles	`132087` cycles	`0.99`
`ML-KEM-1024 keypair`	`124958` cycles	`123924` cycles	`1.01`
`ML-KEM-1024 encaps`	`145034` cycles	`145681` cycles	`1.00`
`ML-KEM-1024 decaps`	`182563` cycles	`183541` cycles	`0.99`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a) (no-opt)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`45739` cycles	`45741` cycles	`1.00`
`ML-KEM-512 encaps`	`56914` cycles	`56848` cycles	`1.00`
`ML-KEM-512 decaps`	`76257` cycles	`76239` cycles	`1.00`
`ML-KEM-768 keypair`	`74530` cycles	`74437` cycles	`1.00`
`ML-KEM-768 encaps`	`88560` cycles	`88581` cycles	`1.00`
`ML-KEM-768 decaps`	`114738` cycles	`114627` cycles	`1.00`
`ML-KEM-1024 keypair`	`109390` cycles	`109371` cycles	`1.00`
`ML-KEM-1024 encaps`	`127333` cycles	`127232` cycles	`1.00`
`ML-KEM-1024 decaps`	`159950` cycles	`159932` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i) (no-opt)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`56624` cycles	`56562` cycles	`1.00`
`ML-KEM-512 encaps`	`69535` cycles	`69509` cycles	`1.00`
`ML-KEM-512 decaps`	`91408` cycles	`91299` cycles	`1.00`
`ML-KEM-768 keypair`	`91934` cycles	`91904` cycles	`1.00`
`ML-KEM-768 encaps`	`107832` cycles	`107780` cycles	`1.00`
`ML-KEM-768 decaps`	`136413` cycles	`136341` cycles	`1.00`
`ML-KEM-1024 keypair`	`134857` cycles	`134603` cycles	`1.00`
`ML-KEM-1024 encaps`	`155279` cycles	`155122` cycles	`1.00`
`ML-KEM-1024 decaps`	`191623` cycles	`191607` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`18113` cycles	`18110` cycles	`1.00`
`ML-KEM-512 encaps`	`22173` cycles	`22177` cycles	`1.00`
`ML-KEM-512 decaps`	`28832` cycles	`28840` cycles	`1.00`
`ML-KEM-768 keypair`	`30563` cycles	`30567` cycles	`1.00`
`ML-KEM-768 encaps`	`33627` cycles	`33632` cycles	`1.00`
`ML-KEM-768 decaps`	`43167` cycles	`43154` cycles	`1.00`
`ML-KEM-1024 keypair`	`44179` cycles	`44163` cycles	`1.00`
`ML-KEM-1024 encaps`	`49660` cycles	`49650` cycles	`1.00`
`ML-KEM-1024 decaps`	`62622` cycles	`62646` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`29040` cycles	`28985` cycles	`1.00`
`ML-KEM-512 encaps`	`35390` cycles	`35422` cycles	`1.00`
`ML-KEM-512 decaps`	`45890` cycles	`45886` cycles	`1.00`
`ML-KEM-768 keypair`	`49384` cycles	`49378` cycles	`1.00`
`ML-KEM-768 encaps`	`55589` cycles	`55562` cycles	`1.00`
`ML-KEM-768 decaps`	`70353` cycles	`70312` cycles	`1.00`
`ML-KEM-1024 keypair`	`72070` cycles	`71972` cycles	`1.00`
`ML-KEM-1024 encaps`	`80894` cycles	`80777` cycles	`1.00`
`ML-KEM-1024 decaps`	`100762` cycles	`100635` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3 (no-opt)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`45384` cycles	`45405` cycles	`1.00`
`ML-KEM-512 encaps`	`54366` cycles	`54334` cycles	`1.00`
`ML-KEM-512 decaps`	`71277` cycles	`71382` cycles	`1.00`
`ML-KEM-768 keypair`	`74884` cycles	`74871` cycles	`1.00`
`ML-KEM-768 encaps`	`86146` cycles	`86175` cycles	`1.00`
`ML-KEM-768 decaps`	`108635` cycles	`108657` cycles	`1.00`
`ML-KEM-1024 keypair`	`111050` cycles	`111029` cycles	`1.00`
`ML-KEM-1024 encaps`	`125879` cycles	`125882` cycles	`1.00`
`ML-KEM-1024 decaps`	`154631` cycles	`154641` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4 (no-opt)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`41956` cycles	`42045` cycles	`1.00`
`ML-KEM-512 encaps`	`50126` cycles	`50179` cycles	`1.00`
`ML-KEM-512 decaps`	`66047` cycles	`66084` cycles	`1.00`
`ML-KEM-768 keypair`	`69030` cycles	`69056` cycles	`1.00`
`ML-KEM-768 encaps`	`79878` cycles	`79781` cycles	`1.00`
`ML-KEM-768 decaps`	`101139` cycles	`101043` cycles	`1.00`
`ML-KEM-1024 keypair`	`102241` cycles	`102197` cycles	`1.00`
`ML-KEM-1024 encaps`	`117256` cycles	`117160` cycles	`1.00`
`ML-KEM-1024 decaps`	`143414` cycles	`143752` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2 (no-opt)

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`71349` cycles	`71284` cycles	`1.00`
`ML-KEM-512 encaps`	`85013` cycles	`85133` cycles	`1.00`
`ML-KEM-512 decaps`	`112758` cycles	`112718` cycles	`1.00`
`ML-KEM-768 keypair`	`117874` cycles	`117640` cycles	`1.00`
`ML-KEM-768 encaps`	`135529` cycles	`135357` cycles	`1.00`
`ML-KEM-768 decaps`	`172023` cycles	`172079` cycles	`1.00`
`ML-KEM-1024 keypair`	`175441` cycles	`175297` cycles	`1.00`
`ML-KEM-1024 encaps`	`197547` cycles	`197386` cycles	`1.00`
`ML-KEM-1024 decaps`	`243817` cycles	`243542` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A55 (Snapdragon 888) benchmarks

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`58330` cycles	`58346` cycles	`1.00`
`ML-KEM-512 encaps`	`65830` cycles	`65829` cycles	`1.00`
`ML-KEM-512 decaps`	`84526` cycles	`84544` cycles	`1.00`
`ML-KEM-768 keypair`	`98948` cycles	`98950` cycles	`1.00`
`ML-KEM-768 encaps`	`110220` cycles	`110344` cycles	`1.00`
`ML-KEM-768 decaps`	`137024` cycles	`137016` cycles	`1.00`
`ML-KEM-1024 keypair`	`150121` cycles	`150196` cycles	`1.00`
`ML-KEM-1024 encaps`	`166786` cycles	`166391` cycles	`1.00`
`ML-KEM-1024 decaps`	`202461` cycles	`202386` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Bananapi bpi-f3 benchmarks

Benchmark suite	Current: `e05e7ce`	Previous: `a08df02`	Ratio
`ML-KEM-512 keypair`	`334581` cycles	`334519` cycles	`1.00`
`ML-KEM-512 encaps`	`445897` cycles	`445835` cycles	`1.00`
`ML-KEM-512 decaps`	`594172` cycles	`593916` cycles	`1.00`
`ML-KEM-768 keypair`	`554993` cycles	`554910` cycles	`1.00`
`ML-KEM-768 encaps`	`697015` cycles	`696817` cycles	`1.00`
`ML-KEM-768 decaps`	`889548` cycles	`889198` cycles	`1.00`
`ML-KEM-1024 keypair`	`819460` cycles	`819196` cycles	`1.00`
`ML-KEM-1024 encaps`	`997449` cycles	`996119` cycles	`1.00`
`ML-KEM-1024 decaps`	`1228626` cycles	`1228975` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

hanno-becker requested a review from mkannwischer January 13, 2025 19:33

hanno-becker marked this pull request as ready for review January 13, 2025 19:33

hanno-becker requested a review from a team January 13, 2025 19:33

mkannwischer added the benchmark this PR should be benchmarked in CI label Jan 14, 2025

mkannwischer approved these changes Jan 14, 2025

View reviewed changes

oqs-bot reviewed Jan 14, 2025

View reviewed changes

mkannwischer merged commit 8383b16 into main Jan 14, 2025
156 checks passed

mkannwischer deleted the test_opt_fix branch January 14, 2025 04:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `tests bench [--opt=ALL]` so it runs opt and non-opt benchmarks #657

Fix `tests bench [--opt=ALL]` so it runs opt and non-opt benchmarks #657

hanno-becker commented Jan 13, 2025 •

edited by mkannwischer

Loading

mkannwischer left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

Fix tests bench [--opt=ALL] so it runs opt and non-opt benchmarks #657

Fix tests bench [--opt=ALL] so it runs opt and non-opt benchmarks #657

Conversation

hanno-becker commented Jan 13, 2025 • edited by mkannwischer Loading

mkannwischer left a comment

Choose a reason for hiding this comment

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 4th gen (c7i)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 3rd gen (c6a)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 3rd gen (c6i)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 4th gen (c7a)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 4th gen (c7i) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton3

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 3rd gen (c6a) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 4th gen (c7a) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 3rd gen (c6i) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton4

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton2

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton3 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton4 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton2 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A55 (Snapdragon 888) benchmarks

oqs-bot left a comment

Choose a reason for hiding this comment

Bananapi bpi-f3 benchmarks

Fix `tests bench [--opt=ALL]` so it runs opt and non-opt benchmarks #657

Fix `tests bench [--opt=ALL]` so it runs opt and non-opt benchmarks #657

hanno-becker commented Jan 13, 2025 •

edited by mkannwischer

Loading