Benchmark #1463
Merged
Benchmark #1463
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1463 +/- ##
=========================================
Coverage 99.92% 99.93%
=========================================
Files 61 61
Lines 21319 23213 +1894
=========================================
+ Hits 21304 23198 +1894
Misses 15 15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
well well, the output for this leaves a LOT to be sifted through. May be wise to trim benchmarks down to a few specific cases. Currently it is running a matrix of starting # and lengths for both directions of baseline to antnum conversions |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Benchmark
| Benchmark suite | Current: efda0e3 | Previous: 849a550 | Ratio |
|---|---|---|---|
tests/utils/test_bls.py::test_bls_to_ant[min=0-len=1] |
60244.99014708503 iter/sec (stddev: 0.0000015673352579161238) |
58602.41845976454 iter/sec (stddev: 0.000002437645243754866) |
0.97 |
tests/utils/test_bls.py::test_bls_to_ant[min=0-len=10] |
60255.32946159182 iter/sec (stddev: 0.000001969652993043119) |
59073.18450223305 iter/sec (stddev: 0.000002865792787814884) |
0.98 |
tests/utils/test_bls.py::test_bls_to_ant[min=0-len=100] |
56734.218702043116 iter/sec (stddev: 0.000003682987090303425) |
57325.16051824902 iter/sec (stddev: 0.000002244330131156411) |
1.01 |
tests/utils/test_bls.py::test_bls_to_ant[min=0-len=1000] |
43758.506153419534 iter/sec (stddev: 0.0000037899351973204605) |
43792.36143695388 iter/sec (stddev: 0.0000026353557898239846) |
1.00 |
tests/utils/test_bls.py::test_bls_to_ant[min=0-len=10000] |
15634.783479844738 iter/sec (stddev: 0.000005499737408328288) |
15364.801179736245 iter/sec (stddev: 0.000008887543704586156) |
0.98 |
tests/utils/test_bls.py::test_bls_to_ant[min=0-len=100000] |
1313.9487138399572 iter/sec (stddev: 0.00003325822231337757) |
1316.1761383287135 iter/sec (stddev: 0.000037645614400192726) |
1.00 |
tests/utils/test_bls.py::test_bls_to_ant[min=0-len=1000000] |
94.3168868588436 iter/sec (stddev: 0.00044271343086347664) |
95.12160298170191 iter/sec (stddev: 0.0007268205237581817) |
1.01 |
tests/utils/test_bls.py::test_bls_to_ant[min=65536-len=1] |
59645.48469277859 iter/sec (stddev: 0.000002657087456168566) |
58566.76989027288 iter/sec (stddev: 0.0000032416736519115015) |
0.98 |
tests/utils/test_bls.py::test_bls_to_ant[min=65536-len=10] |
60187.92720143719 iter/sec (stddev: 0.000002167961838505244) |
58790.34284464189 iter/sec (stddev: 0.0000030823844993773287) |
0.98 |
tests/utils/test_bls.py::test_bls_to_ant[min=65536-len=100] |
57872.81982031443 iter/sec (stddev: 0.0000028090520749859158) |
57196.1518977742 iter/sec (stddev: 0.000002478449918043021) |
0.99 |
tests/utils/test_bls.py::test_bls_to_ant[min=65536-len=1000] |
44174.507022083744 iter/sec (stddev: 0.000002887160255428934) |
43042.56497845993 iter/sec (stddev: 0.000003969068049139489) |
0.97 |
tests/utils/test_bls.py::test_bls_to_ant[min=65536-len=10000] |
15179.080677324486 iter/sec (stddev: 0.000005776806916725941) |
15119.25061525153 iter/sec (stddev: 0.000006357983812662383) |
1.00 |
tests/utils/test_bls.py::test_bls_to_ant[min=65536-len=100000] |
1314.2404021762006 iter/sec (stddev: 0.00003917142276063729) |
1329.3537198317958 iter/sec (stddev: 0.00003493912359949524) |
1.01 |
tests/utils/test_bls.py::test_bls_to_ant[min=65536-len=1000000] |
93.5774256812527 iter/sec (stddev: 0.00044259424534055206) |
93.34331829579715 iter/sec (stddev: 0.0008473740625094813) |
1.00 |
tests/utils/test_bls.py::test_bls_to_ant[min=4259840-len=1] |
61006.92045692254 iter/sec (stddev: 0.00000226178718107033) |
59176.46678625062 iter/sec (stddev: 0.000002718178855318947) |
0.97 |
tests/utils/test_bls.py::test_bls_to_ant[min=4259840-len=10] |
59614.04791845897 iter/sec (stddev: 0.0000029529385595137157) |
58592.201786167 iter/sec (stddev: 0.000004816910764919944) |
0.98 |
tests/utils/test_bls.py::test_bls_to_ant[min=4259840-len=100] |
56878.35671485366 iter/sec (stddev: 0.0000023031360065513934) |
56019.30573379863 iter/sec (stddev: 0.000001956703383808175) |
0.98 |
tests/utils/test_bls.py::test_bls_to_ant[min=4259840-len=1000] |
38424.81461735033 iter/sec (stddev: 0.0000033602695800629727) |
37253.24461024788 iter/sec (stddev: 0.000005243736818684219) |
0.97 |
tests/utils/test_bls.py::test_bls_to_ant[min=4259840-len=10000] |
10032.234721447501 iter/sec (stddev: 0.000006249999367584577) |
9830.031321784445 iter/sec (stddev: 0.000015598747199492704) |
0.98 |
tests/utils/test_bls.py::test_bls_to_ant[min=4259840-len=100000] |
972.3595161033617 iter/sec (stddev: 0.00003952422502620619) |
974.4706202928901 iter/sec (stddev: 0.000047215186090252536) |
1.00 |
tests/utils/test_bls.py::test_bls_to_ant[min=4259840-len=1000000] |
74.75421880769944 iter/sec (stddev: 0.00042492941330095693) |
76.2721936394971 iter/sec (stddev: 0.00023254788881437631) |
1.02 |
tests/utils/test_bls.py::test_ants_to_bls[min=0-len=1] |
25014.448578363448 iter/sec (stddev: 0.000004228300623383479) |
24168.43372870323 iter/sec (stddev: 0.000007667222628886107) |
0.97 |
tests/utils/test_bls.py::test_ants_to_bls[min=0-len=10] |
22825.726158104197 iter/sec (stddev: 0.000003840197471344839) |
21683.341511937982 iter/sec (stddev: 0.00000834977822712593) |
0.95 |
tests/utils/test_bls.py::test_ants_to_bls[min=0-len=100] |
21762.50541314487 iter/sec (stddev: 0.000003957377923452244) |
21027.32563259485 iter/sec (stddev: 0.000006499201139750355) |
0.97 |
tests/utils/test_bls.py::test_ants_to_bls[min=0-len=1000] |
12905.49479403442 iter/sec (stddev: 0.0000072949743170999824) |
12757.973621497646 iter/sec (stddev: 0.000006892029733863968) |
0.99 |
tests/utils/test_bls.py::test_ants_to_bls[min=0-len=10000] |
2775.3408825616552 iter/sec (stddev: 0.00001911556401114875) |
2699.944104116573 iter/sec (stddev: 0.000021116767682607908) |
0.97 |
tests/utils/test_bls.py::test_ants_to_bls[min=0-len=100000] |
278.9362035079354 iter/sec (stddev: 0.0000454585071301089) |
277.7137460195779 iter/sec (stddev: 0.0000962358366136919) |
1.00 |
tests/utils/test_bls.py::test_ants_to_bls[min=0-len=1000000] |
21.4748811731332 iter/sec (stddev: 0.0005600237319697404) |
23.03002818204356 iter/sec (stddev: 0.0004176626779990228) |
1.07 |
tests/utils/test_bls.py::test_ants_to_bls[min=65536-len=1] |
24998.55472974417 iter/sec (stddev: 0.000005237838472162775) |
24558.23642667754 iter/sec (stddev: 0.00000505115341628858) |
0.98 |
tests/utils/test_bls.py::test_ants_to_bls[min=65536-len=10] |
22831.796673752062 iter/sec (stddev: 0.000005097605907761957) |
22345.067275315883 iter/sec (stddev: 0.000005073479934121733) |
0.98 |
tests/utils/test_bls.py::test_ants_to_bls[min=65536-len=100] |
21750.064176691627 iter/sec (stddev: 0.000004532073580879082) |
21404.975987902595 iter/sec (stddev: 0.000004532238514832773) |
0.98 |
tests/utils/test_bls.py::test_ants_to_bls[min=65536-len=1000] |
13941.926029366432 iter/sec (stddev: 0.000005445963583219551) |
13828.286431763841 iter/sec (stddev: 0.000007097502134078503) |
0.99 |
tests/utils/test_bls.py::test_ants_to_bls[min=65536-len=10000] |
2787.6526008831493 iter/sec (stddev: 0.00001944928624719588) |
2669.685663902722 iter/sec (stddev: 0.000027777668992840686) |
0.96 |
tests/utils/test_bls.py::test_ants_to_bls[min=65536-len=100000] |
232.01922298871497 iter/sec (stddev: 0.00007354849303121135) |
251.77831548765894 iter/sec (stddev: 0.0003111452525703316) |
1.09 |
tests/utils/test_bls.py::test_ants_to_bls[min=65536-len=1000000] |
22.1219871776927 iter/sec (stddev: 0.00047151596951243614) |
23.55948596069528 iter/sec (stddev: 0.00039148236679147725) |
1.06 |
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=1] |
24625.2242161493 iter/sec (stddev: 0.000010913870722968013) |
24706.49785754261 iter/sec (stddev: 0.000005902091491244575) |
1.00 |
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=10] |
22786.67926721286 iter/sec (stddev: 0.000023035838533233804) |
22222.93552120562 iter/sec (stddev: 0.0000054768029738922055) |
0.98 |
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=100] |
21811.95390569394 iter/sec (stddev: 0.0000048663760156835704) |
21425.029807343908 iter/sec (stddev: 0.00000535294164644571) |
0.98 |
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=1000] |
14065.045668700419 iter/sec (stddev: 0.000004480359676236143) |
13885.838799480474 iter/sec (stddev: 0.000006774776746467678) |
0.99 |
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=10000] |
3111.918718433128 iter/sec (stddev: 0.00001595486101246816) |
3086.2751820836293 iter/sec (stddev: 0.00002133461895799764) |
0.99 |
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=100000] |
260.4935576746864 iter/sec (stddev: 0.0003570399907874279) |
293.71835184745964 iter/sec (stddev: 0.00009768114106972216) |
1.13 |
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=1000000] |
18.620128888535003 iter/sec (stddev: 0.0007581896146330588) |
19.34894102762067 iter/sec (stddev: 0.00023659786063420261) |
1.04 |
tests/uvdata/test_mwa_corr_fits.py::test_corr_fits_select_on_read |
10.687299138893682 iter/sec (stddev: 0.007944098546795042) |
10.63565970072557 iter/sec (stddev: 0.009238993496421204) |
1.00 |
tests/uvdata/test_mwa_corr_fits.py::test_van_vleck[cheby=True] |
9.425324606216304 iter/sec (stddev: 0.0009569091924824441) |
9.16052300923503 iter/sec (stddev: 0.001490273648083755) |
0.97 |
tests/uvdata/test_mwa_corr_fits.py::test_van_vleck[cheby=False] |
0.08331622212535357 iter/sec (stddev: 0.024116725687899257) |
0.08299149895668465 iter/sec (stddev: 0.03641371749024679) |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
bhazelton
reviewed
Aug 13, 2024
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: c8100fe | Previous: 13eea2c | Ratio |
|---|---|---|---|
tests/utils/test_bls.py::test_ants_to_bls[min=4259840-len=10000] |
959.3935052649105 iter/sec (stddev: 0.00478042998546704) |
2518.406883281963 iter/sec (stddev: 0.00005636511610640517) |
2.62 |
This comment was automatically generated by workflow using github-action-benchmark.
options don't seem to affect the results
|
should we add a check that only stores the benchmark data when we run on main? |
|
@bhazelton I think this is ready for a look now. We'll only store the benchmark results when executed from the main branch now. interestingly the baseline/antenna-number conversion triggered a failure this last time 😆 |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds CI benchmarking so we can monitor our performance sensitive areas.
Motivation and Context
Because manually benchmarking is so 2002
Types of changes