Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: migrate spec-test-utils and spec tests to vitest #6222

Merged
merged 19 commits into from
Jan 15, 2024
Merged

test: migrate spec-test-utils and spec tests to vitest #6222

merged 19 commits into from
Jan 15, 2024

Conversation

nazarhussain
Copy link
Contributor

Motivation

Consolidate the testing frameworks and migrate to vitest.

Description

Migrate spec-test-utils and spec tests to vitest.

Steps to test or reproduce

  • Run all tests

@nazarhussain nazarhussain requested a review from a team as a code owner December 21, 2023 15:26
@nazarhussain nazarhussain self-assigned this Dec 21, 2023
nazarhussain
nazarhussain commented Dec 21, 2023
View reviewed changes
packages/spec-test-util/package.json Outdated
Comment on lines 12 to 19
"exports": {
".": {
"import": "./lib/index.js"
},
"./download": {
"import": "./lib/downloadTests.js"
}
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any code which imports vitest should be run via vitest binary. For download script we need to run it directly with nodejs so why have to extract that part as separate export.

nazarhussain
nazarhussain commented Dec 21, 2023
View reviewed changes
packages/spec-test-util/package.json
@@ -26,12 +33,13 @@
"build": "tsc -p tsconfig.build.json",
"build:release": "yarn clean && yarn build",
"build:watch": "yarn run build --watch",
"check-build": "node -e \"(async function() { await import('./lib/index.js') })()\"",
"check-build": "node -e \"(async function() { await import('./lib/downloadTests.js') })()\"",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in above comment.

@codecov Codecov
Copy link

codecov bot commented Dec 21, 2023 •
edited
Loading

Codecov Report

Merging #6222 (a7764f0) into unstable (0b2d35a) will not change coverage.
Report is 7 commits behind head on unstable.
The diff coverage is n/a.

Additional details and impacted files 
@@            Coverage Diff            @@
##           unstable    #6222   +/-   ##
=========================================
  Coverage     80.38%   80.38%           
=========================================
  Files           202      202           
  Lines         19620    19620           
  Branches       1176     1176           
=========================================
  Hits          15771    15771           
  Misses         3821     3821           
  Partials         28       28

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 21, 2023 •
edited
Loading

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: 72a1755 Previous: 098d35a Ratio
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 29.450 us/op 7.0010 us/op 4.21
Full benchmark results
Benchmark suite Current: 72a1755 Previous: 098d35a Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.2291 ms/op 888.87 us/op 1.38
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 220.64 us/op 81.806 us/op 2.70
BLS verify - blst-native 1.9425 ms/op 1.2765 ms/op 1.52
BLS verifyMultipleSignatures 3 - blst-native 4.0358 ms/op 2.6851 ms/op 1.50
BLS verifyMultipleSignatures 8 - blst-native 8.8351 ms/op 5.8816 ms/op 1.50
BLS verifyMultipleSignatures 32 - blst-native 32.591 ms/op 21.530 ms/op 1.51
BLS verifyMultipleSignatures 64 - blst-native 63.071 ms/op 42.411 ms/op 1.49
BLS verifyMultipleSignatures 128 - blst-native 123.64 ms/op 84.174 ms/op 1.47
BLS deserializing 10000 signatures 1.2852 s/op 897.12 ms/op 1.43
BLS deserializing 100000 signatures 11.458 s/op 9.0524 s/op 1.27
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.6072 ms/op 1.2609 ms/op 1.27
BLS verifyMultipleSignatures - same message - 8 - blst-native 2.1898 ms/op 1.5241 ms/op 1.44
BLS verifyMultipleSignatures - same message - 32 - blst-native 3.1815 ms/op 2.1958 ms/op 1.45
BLS verifyMultipleSignatures - same message - 64 - blst-native 5.1089 ms/op 4.2555 ms/op 1.20
BLS verifyMultipleSignatures - same message - 128 - blst-native 8.4532 ms/op 7.3021 ms/op 1.16
BLS aggregatePubkeys 32 - blst-native 39.668 us/op 26.835 us/op 1.48
BLS aggregatePubkeys 128 - blst-native 117.76 us/op 96.919 us/op 1.21
getAttestationsForBlock 77.517 ms/op 43.893 ms/op 1.77
getSlashingsAndExits - default max 192.14 us/op 167.21 us/op 1.15
getSlashingsAndExits - 2k 598.77 us/op 362.90 us/op 1.65
proposeBlockBody type=full, size=empty 7.9213 ms/op 4.9525 ms/op 1.60
isKnown best case - 1 super set check 712.00 ns/op 322.00 ns/op 2.21
isKnown normal case - 2 super set checks 852.00 ns/op 292.00 ns/op 2.92
isKnown worse case - 16 super set checks 770.00 ns/op 280.00 ns/op 2.75
CheckpointStateCache - add get delete 7.5340 us/op 5.5280 us/op 1.36
validate api signedAggregateAndProof - struct 3.2107 ms/op 2.7344 ms/op 1.17
validate gossip signedAggregateAndProof - struct 3.2900 ms/op 2.7392 ms/op 1.20
validate gossip attestation - vc 640000 1.6238 ms/op 1.3155 ms/op 1.23
batch validate gossip attestation - vc 640000 - chunk 32 196.54 us/op 156.51 us/op 1.26
batch validate gossip attestation - vc 640000 - chunk 64 171.62 us/op 136.96 us/op 1.25
batch validate gossip attestation - vc 640000 - chunk 128 176.82 us/op 126.47 us/op 1.40
batch validate gossip attestation - vc 640000 - chunk 256 162.18 us/op 129.92 us/op 1.25
pickEth1Vote - no votes 1.6191 ms/op 1.1328 ms/op 1.43
pickEth1Vote - max votes 14.886 ms/op 10.212 ms/op 1.46
pickEth1Vote - Eth1Data hashTreeRoot value x2048 22.844 ms/op 21.411 ms/op 1.07
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 43.226 ms/op 27.950 ms/op 1.55
pickEth1Vote - Eth1Data fastSerialize value x2048 840.64 us/op 574.72 us/op 1.46
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.3608 ms/op 6.7798 ms/op 0.94
bytes32 toHexString 850.00 ns/op 492.00 ns/op 1.73
bytes32 Buffer.toString(hex) 328.00 ns/op 281.00 ns/op 1.17
bytes32 Buffer.toString(hex) from Uint8Array 640.00 ns/op 417.00 ns/op 1.53
bytes32 Buffer.toString(hex) + 0x 348.00 ns/op 279.00 ns/op 1.25
Object access 1 prop 0.23500 ns/op 0.15500 ns/op 1.52
Map access 1 prop 0.16600 ns/op 0.13100 ns/op 1.27
Object get x1000 8.7840 ns/op 7.6080 ns/op 1.15
Map get x1000 0.94500 ns/op 0.71600 ns/op 1.32
Object set x1000 77.099 ns/op 48.308 ns/op 1.60
Map set x1000 56.791 ns/op 36.982 ns/op 1.54
Return object 10000 times 0.26870 ns/op 0.22620 ns/op 1.19
Throw Error 10000 times 4.3312 us/op 3.6822 us/op 1.18
fastMsgIdFn sha256 / 200 bytes 3.7830 us/op 3.1060 us/op 1.22
fastMsgIdFn h32 xxhash / 200 bytes 376.00 ns/op 265.00 ns/op 1.42
fastMsgIdFn h64 xxhash / 200 bytes 413.00 ns/op 323.00 ns/op 1.28
fastMsgIdFn sha256 / 1000 bytes 12.730 us/op 10.945 us/op 1.16
fastMsgIdFn h32 xxhash / 1000 bytes 519.00 ns/op 395.00 ns/op 1.31
fastMsgIdFn h64 xxhash / 1000 bytes 496.00 ns/op 392.00 ns/op 1.27
fastMsgIdFn sha256 / 10000 bytes 112.84 us/op 99.916 us/op 1.13
fastMsgIdFn h32 xxhash / 10000 bytes 2.2030 us/op 1.8590 us/op 1.19
fastMsgIdFn h64 xxhash / 10000 bytes 1.5370 us/op 1.2660 us/op 1.21
send data - 1000 256B messages 23.165 ms/op 18.371 ms/op 1.26
send data - 1000 512B messages 32.031 ms/op 27.519 ms/op 1.16
send data - 1000 1024B messages 49.463 ms/op 39.244 ms/op 1.26
send data - 1000 1200B messages 44.456 ms/op 35.531 ms/op 1.25
send data - 1000 2048B messages 53.239 ms/op 42.203 ms/op 1.26
send data - 1000 4096B messages 50.167 ms/op 38.492 ms/op 1.30
send data - 1000 16384B messages 118.37 ms/op 115.46 ms/op 1.03
send data - 1000 65536B messages 492.01 ms/op 491.72 ms/op 1.00
enrSubnets - fastDeserialize 64 bits 1.6540 us/op 1.2300 us/op 1.34
enrSubnets - ssz BitVector 64 bits 552.00 ns/op 412.00 ns/op 1.34
enrSubnets - fastDeserialize 4 bits 256.00 ns/op 164.00 ns/op 1.56
enrSubnets - ssz BitVector 4 bits 600.00 ns/op 411.00 ns/op 1.46
prioritizePeers score -10:0 att 32-0.1 sync 2-0 122.95 us/op 99.463 us/op 1.24
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 170.38 us/op 124.24 us/op 1.37
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 213.16 us/op 159.19 us/op 1.34
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 388.19 us/op 282.49 us/op 1.37
prioritizePeers score 0:0 att 64-1 sync 4-1 425.26 us/op 332.88 us/op 1.28
array of 16000 items push then shift 1.8306 us/op 1.5714 us/op 1.16
LinkedList of 16000 items push then shift 10.096 ns/op 8.8190 ns/op 1.14
array of 16000 items push then pop 130.51 ns/op 70.913 ns/op 1.84
LinkedList of 16000 items push then pop 10.079 ns/op 8.6040 ns/op 1.17
array of 24000 items push then shift 2.7753 us/op 2.3384 us/op 1.19
LinkedList of 24000 items push then shift 10.421 ns/op 8.7580 ns/op 1.19
array of 24000 items push then pop 163.24 ns/op 97.806 ns/op 1.67
LinkedList of 24000 items push then pop 9.9030 ns/op 8.5670 ns/op 1.16
intersect bitArray bitLen 8 7.5410 ns/op 6.2680 ns/op 1.20
intersect array and set length 8 90.946 ns/op 60.829 ns/op 1.50
intersect bitArray bitLen 128 36.998 ns/op 33.197 ns/op 1.11
intersect array and set length 128 1.2463 us/op 838.91 ns/op 1.49
bitArray.getTrueBitIndexes() bitLen 128 2.0330 us/op 1.3520 us/op 1.50
bitArray.getTrueBitIndexes() bitLen 248 3.2570 us/op 2.4010 us/op 1.36
bitArray.getTrueBitIndexes() bitLen 512 6.4270 us/op 4.5520 us/op 1.41
Buffer.concat 32 items 1.0300 us/op 963.00 ns/op 1.07
Uint8Array.set 32 items 1.8740 us/op 2.0480 us/op 0.92
Set add up to 64 items then delete first 5.2601 us/op 4.1959 us/op 1.25
OrderedSet add up to 64 items then delete first 6.7833 us/op 5.3522 us/op 1.27
Set add up to 64 items then delete last 5.7339 us/op 4.5169 us/op 1.27
OrderedSet add up to 64 items then delete last 7.2707 us/op 5.7226 us/op 1.27
Set add up to 64 items then delete middle 5.6125 us/op 4.4896 us/op 1.25
OrderedSet add up to 64 items then delete middle 9.3040 us/op 6.8868 us/op 1.35
Set add up to 128 items then delete first 11.279 us/op 9.1514 us/op 1.23
OrderedSet add up to 128 items then delete first 15.291 us/op 12.073 us/op 1.27
Set add up to 128 items then delete last 11.682 us/op 9.0686 us/op 1.29
OrderedSet add up to 128 items then delete last 15.981 us/op 11.341 us/op 1.41
Set add up to 128 items then delete middle 11.470 us/op 8.9844 us/op 1.28
OrderedSet add up to 128 items then delete middle 21.785 us/op 16.622 us/op 1.31
Set add up to 256 items then delete first 23.034 us/op 18.215 us/op 1.26
OrderedSet add up to 256 items then delete first 31.473 us/op 24.649 us/op 1.28
Set add up to 256 items then delete last 23.546 us/op 17.833 us/op 1.32
OrderedSet add up to 256 items then delete last 33.019 us/op 22.833 us/op 1.45
Set add up to 256 items then delete middle 23.782 us/op 17.903 us/op 1.33
OrderedSet add up to 256 items then delete middle 58.225 us/op 44.007 us/op 1.32
transfer serialized Status (84 B) 2.0230 us/op 1.7650 us/op 1.15
copy serialized Status (84 B) 1.8070 us/op 1.4630 us/op 1.24
transfer serialized SignedVoluntaryExit (112 B) 2.1610 us/op 1.9200 us/op 1.13
copy serialized SignedVoluntaryExit (112 B) 1.8890 us/op 1.5010 us/op 1.26
transfer serialized ProposerSlashing (416 B) 2.5340 us/op 2.3260 us/op 1.09
copy serialized ProposerSlashing (416 B) 2.8210 us/op 2.1790 us/op 1.29
transfer serialized Attestation (485 B) 2.4520 us/op 2.4240 us/op 1.01
copy serialized Attestation (485 B) 2.6230 us/op 2.1480 us/op 1.22
transfer serialized AttesterSlashing (33232 B) 2.5890 us/op 2.7670 us/op 0.94
copy serialized AttesterSlashing (33232 B) 9.4820 us/op 5.6870 us/op 1.67
transfer serialized Small SignedBeaconBlock (128000 B) 3.0640 us/op 3.1460 us/op 0.97
copy serialized Small SignedBeaconBlock (128000 B) 31.515 us/op 13.742 us/op 2.29
transfer serialized Avg SignedBeaconBlock (200000 B) 3.3130 us/op 3.1660 us/op 1.05
copy serialized Avg SignedBeaconBlock (200000 B) 44.947 us/op 61.172 us/op 0.73
transfer serialized BlobsSidecar (524380 B) 3.7140 us/op 3.4140 us/op 1.09
copy serialized BlobsSidecar (524380 B) 123.97 us/op 143.30 us/op 0.87
transfer serialized Big SignedBeaconBlock (1000000 B) 3.9880 us/op 3.3340 us/op 1.20
copy serialized Big SignedBeaconBlock (1000000 B) 194.13 us/op 289.61 us/op 0.67
pass gossip attestations to forkchoice per slot 4.4268 ms/op 4.0072 ms/op 1.10
forkChoice updateHead vc 100000 bc 64 eq 0 753.87 us/op 672.26 us/op 1.12
forkChoice updateHead vc 600000 bc 64 eq 0 4.6223 ms/op 4.1098 ms/op 1.12
forkChoice updateHead vc 1000000 bc 64 eq 0 8.3334 ms/op 6.9299 ms/op 1.20
forkChoice updateHead vc 600000 bc 320 eq 0 7.9073 ms/op 4.0727 ms/op 1.94
forkChoice updateHead vc 600000 bc 1200 eq 0 5.2839 ms/op 4.3032 ms/op 1.23
forkChoice updateHead vc 600000 bc 7200 eq 0 6.1751 ms/op 5.1962 ms/op 1.19
forkChoice updateHead vc 600000 bc 64 eq 1000 13.021 ms/op 11.016 ms/op 1.18
forkChoice updateHead vc 600000 bc 64 eq 10000 15.175 ms/op 11.799 ms/op 1.29
forkChoice updateHead vc 600000 bc 64 eq 300000 28.595 ms/op 15.713 ms/op 1.82
computeDeltas 500000 validators 300 proto nodes 7.6530 ms/op 6.4179 ms/op 1.19
computeDeltas 500000 validators 1200 proto nodes 7.7200 ms/op 6.3016 ms/op 1.23
computeDeltas 500000 validators 7200 proto nodes 7.6798 ms/op 6.1984 ms/op 1.24
computeDeltas 750000 validators 300 proto nodes 12.027 ms/op 9.5143 ms/op 1.26
computeDeltas 750000 validators 1200 proto nodes 11.565 ms/op 9.4484 ms/op 1.22
computeDeltas 750000 validators 7200 proto nodes 10.875 ms/op 9.6244 ms/op 1.13
computeDeltas 1400000 validators 300 proto nodes 21.791 ms/op 17.789 ms/op 1.22
computeDeltas 1400000 validators 1200 proto nodes 21.303 ms/op 18.136 ms/op 1.17
computeDeltas 1400000 validators 7200 proto nodes 20.672 ms/op 17.790 ms/op 1.16
computeDeltas 2100000 validators 300 proto nodes 31.076 ms/op 27.878 ms/op 1.11
computeDeltas 2100000 validators 1200 proto nodes 30.428 ms/op 27.713 ms/op 1.10
computeDeltas 2100000 validators 7200 proto nodes 31.217 ms/op 27.761 ms/op 1.12
computeProposerBoostScoreFromBalances 500000 validators 4.0991 ms/op 3.6677 ms/op 1.12
computeProposerBoostScoreFromBalances 750000 validators 4.1458 ms/op 3.6746 ms/op 1.13
computeProposerBoostScoreFromBalances 1400000 validators 4.2329 ms/op 3.6649 ms/op 1.15
computeProposerBoostScoreFromBalances 2100000 validators 4.5996 ms/op 3.6749 ms/op 1.25
altair processAttestation - 250000 vs - 7PWei normalcase 4.1702 ms/op 2.0848 ms/op 2.00
altair processAttestation - 250000 vs - 7PWei worstcase 5.2924 ms/op 3.1230 ms/op 1.69
altair processAttestation - setStatus - 1/6 committees join 227.21 us/op 140.95 us/op 1.61
altair processAttestation - setStatus - 1/3 committees join 465.24 us/op 269.85 us/op 1.72
altair processAttestation - setStatus - 1/2 committees join 651.25 us/op 371.26 us/op 1.75
altair processAttestation - setStatus - 2/3 committees join 808.77 us/op 469.22 us/op 1.72
altair processAttestation - setStatus - 4/5 committees join 1.0596 ms/op 664.21 us/op 1.60
altair processAttestation - setStatus - 100% committees join 1.3840 ms/op 761.53 us/op 1.82
altair processBlock - 250000 vs - 7PWei normalcase 16.811 ms/op 9.6696 ms/op 1.74
altair processBlock - 250000 vs - 7PWei normalcase hashState 48.340 ms/op 37.358 ms/op 1.29
altair processBlock - 250000 vs - 7PWei worstcase 56.420 ms/op 37.292 ms/op 1.51
altair processBlock - 250000 vs - 7PWei worstcase hashState 143.22 ms/op 94.421 ms/op 1.52
phase0 processBlock - 250000 vs - 7PWei normalcase 4.2093 ms/op 2.4498 ms/op 1.72
phase0 processBlock - 250000 vs - 7PWei worstcase 44.314 ms/op 30.047 ms/op 1.47
altair processEth1Data - 250000 vs - 7PWei normalcase 887.04 us/op 575.48 us/op 1.54
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 16.682 us/op 10.503 us/op 1.59
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 105.68 us/op 49.470 us/op 2.14
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 42.801 us/op 18.102 us/op 2.36
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 29.450 us/op 7.0010 us/op 4.21
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 296.62 us/op 122.08 us/op 2.43
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 2.1889 ms/op 1.4374 ms/op 1.52
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 3.2786 ms/op 1.5396 ms/op 2.13
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 2.7894 ms/op 1.4982 ms/op 1.86
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 9.1415 ms/op 3.5953 ms/op 2.54
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 4.6513 ms/op 2.2686 ms/op 2.05
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 11.196 ms/op 5.0290 ms/op 2.23
Tree 40 250000 create 844.33 ms/op 353.78 ms/op 2.39
Tree 40 250000 get(125000) 246.36 ns/op 192.00 ns/op 1.28
Tree 40 250000 set(125000) 1.8691 us/op 977.41 ns/op 1.91
Tree 40 250000 toArray() 26.414 ms/op 18.254 ms/op 1.45
Tree 40 250000 iterate all - toArray() + loop 26.745 ms/op 18.390 ms/op 1.45
Tree 40 250000 iterate all - get(i) 85.327 ms/op 64.565 ms/op 1.32
MutableVector 250000 create 19.946 ms/op 12.470 ms/op 1.60
MutableVector 250000 get(125000) 7.6720 ns/op 6.5210 ns/op 1.18
MutableVector 250000 set(125000) 352.60 ns/op 254.73 ns/op 1.38
MutableVector 250000 toArray() 5.6221 ms/op 3.1632 ms/op 1.78
MutableVector 250000 iterate all - toArray() + loop 5.4332 ms/op 3.2663 ms/op 1.66
MutableVector 250000 iterate all - get(i) 1.8237 ms/op 1.5162 ms/op 1.20
Array 250000 create 5.6305 ms/op 2.8420 ms/op 1.98
Array 250000 clone - spread 5.6639 ms/op 1.2547 ms/op 4.51
Array 250000 get(125000) 2.9660 ns/op 1.0420 ns/op 2.85
Array 250000 set(125000) 6.7540 ns/op 4.0770 ns/op 1.66
Array 250000 iterate all - loop 190.57 us/op 163.83 us/op 1.16
effectiveBalanceIncrements clone Uint8Array 300000 93.516 us/op 28.476 us/op 3.28
effectiveBalanceIncrements clone MutableVector 300000 2.3610 us/op 361.00 ns/op 6.54
effectiveBalanceIncrements rw all Uint8Array 300000 225.81 us/op 199.33 us/op 1.13
effectiveBalanceIncrements rw all MutableVector 300000 237.15 ms/op 80.980 ms/op 2.93
phase0 afterProcessEpoch - 250000 vs - 7PWei 134.28 ms/op 113.06 ms/op 1.19
phase0 beforeProcessEpoch - 250000 vs - 7PWei 78.731 ms/op 52.523 ms/op 1.50
altair processEpoch - mainnet_e81889 728.86 ms/op 500.78 ms/op 1.46
mainnet_e81889 - altair beforeProcessEpoch 138.73 ms/op 78.569 ms/op 1.77
mainnet_e81889 - altair processJustificationAndFinalization 40.310 us/op 14.959 us/op 2.69
mainnet_e81889 - altair processInactivityUpdates 15.714 ms/op 5.7191 ms/op 2.75
mainnet_e81889 - altair processRewardsAndPenalties 68.873 ms/op 60.703 ms/op 1.13
mainnet_e81889 - altair processRegistryUpdates 8.8100 us/op 2.4840 us/op 3.55
mainnet_e81889 - altair processSlashings 1.2820 us/op 446.00 ns/op 2.87
mainnet_e81889 - altair processEth1DataReset 2.1120 us/op 484.00 ns/op 4.36
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.0603 ms/op 1.4180 ms/op 1.45
mainnet_e81889 - altair processSlashingsReset 10.498 us/op 4.0430 us/op 2.60
mainnet_e81889 - altair processRandaoMixesReset 13.825 us/op 4.2190 us/op 3.28
mainnet_e81889 - altair processHistoricalRootsUpdate 2.8440 us/op 708.00 ns/op 4.02
mainnet_e81889 - altair processParticipationFlagUpdates 7.9120 us/op 1.8430 us/op 4.29
mainnet_e81889 - altair processSyncCommitteeUpdates 2.2610 us/op 683.00 ns/op 3.31
mainnet_e81889 - altair afterProcessEpoch 143.59 ms/op 116.77 ms/op 1.23
capella processEpoch - mainnet_e217614 3.1030 s/op 2.0613 s/op 1.51
mainnet_e217614 - capella beforeProcessEpoch 664.47 ms/op 482.87 ms/op 1.38
mainnet_e217614 - capella processJustificationAndFinalization 20.147 us/op 19.876 us/op 1.01
mainnet_e217614 - capella processInactivityUpdates 26.063 ms/op 21.052 ms/op 1.24
mainnet_e217614 - capella processRewardsAndPenalties 496.94 ms/op 402.56 ms/op 1.23
mainnet_e217614 - capella processRegistryUpdates 24.670 us/op 16.894 us/op 1.46
mainnet_e217614 - capella processSlashings 821.00 ns/op 540.00 ns/op 1.52
mainnet_e217614 - capella processEth1DataReset 1.2870 us/op 401.00 ns/op 3.21
mainnet_e217614 - capella processEffectiveBalanceUpdates 6.2697 ms/op 4.9395 ms/op 1.27
mainnet_e217614 - capella processSlashingsReset 3.1550 us/op 3.2420 us/op 0.97
mainnet_e217614 - capella processRandaoMixesReset 5.3830 us/op 4.6230 us/op 1.16
mainnet_e217614 - capella processHistoricalRootsUpdate 702.00 ns/op 510.00 ns/op 1.38
mainnet_e217614 - capella processParticipationFlagUpdates 3.6640 us/op 1.6520 us/op 2.22
mainnet_e217614 - capella afterProcessEpoch 338.81 ms/op 315.64 ms/op 1.07
phase0 processEpoch - mainnet_e58758 448.45 ms/op 436.42 ms/op 1.03
mainnet_e58758 - phase0 beforeProcessEpoch 141.16 ms/op 113.07 ms/op 1.25
mainnet_e58758 - phase0 processJustificationAndFinalization 16.753 us/op 14.567 us/op 1.15
mainnet_e58758 - phase0 processRewardsAndPenalties 57.118 ms/op 54.928 ms/op 1.04
mainnet_e58758 - phase0 processRegistryUpdates 10.927 us/op 8.9170 us/op 1.23
mainnet_e58758 - phase0 processSlashings 662.00 ns/op 415.00 ns/op 1.60
mainnet_e58758 - phase0 processEth1DataReset 446.00 ns/op 385.00 ns/op 1.16
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.2259 ms/op 1.1442 ms/op 1.07
mainnet_e58758 - phase0 processSlashingsReset 3.3020 us/op 2.7360 us/op 1.21
mainnet_e58758 - phase0 processRandaoMixesReset 3.8880 us/op 4.1590 us/op 0.93
mainnet_e58758 - phase0 processHistoricalRootsUpdate 500.00 ns/op 391.00 ns/op 1.28
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.9050 us/op 4.9290 us/op 1.00
mainnet_e58758 - phase0 afterProcessEpoch 102.32 ms/op 94.664 ms/op 1.08
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.4534 ms/op 1.3292 ms/op 1.09
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.6307 ms/op 1.9439 ms/op 0.84
altair processInactivityUpdates - 250000 normalcase 27.364 ms/op 30.144 ms/op 0.91
altair processInactivityUpdates - 250000 worstcase 27.167 ms/op 30.633 ms/op 0.89
phase0 processRegistryUpdates - 250000 normalcase 9.9470 us/op 9.5720 us/op 1.04
phase0 processRegistryUpdates - 250000 badcase_full_deposits 456.89 us/op 311.40 us/op 1.47
phase0 processRegistryUpdates - 250000 worstcase 0.5 155.83 ms/op 157.72 ms/op 0.99
altair processRewardsAndPenalties - 250000 normalcase 60.834 ms/op 63.425 ms/op 0.96
altair processRewardsAndPenalties - 250000 worstcase 62.981 ms/op 56.387 ms/op 1.12
phase0 getAttestationDeltas - 250000 normalcase 11.130 ms/op 9.1089 ms/op 1.22
phase0 getAttestationDeltas - 250000 worstcase 12.182 ms/op 8.9035 ms/op 1.37
phase0 processSlashings - 250000 worstcase 112.78 us/op 85.653 us/op 1.32
altair processSyncCommitteeUpdates - 250000 187.89 ms/op 149.30 ms/op 1.26
BeaconState.hashTreeRoot - No change 304.00 ns/op 255.00 ns/op 1.19
BeaconState.hashTreeRoot - 1 full validator 168.05 us/op 147.97 us/op 1.14
BeaconState.hashTreeRoot - 32 full validator 1.7457 ms/op 1.5379 ms/op 1.14
BeaconState.hashTreeRoot - 512 full validator 18.351 ms/op 16.716 ms/op 1.10
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 181.57 us/op 208.62 us/op 0.87
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.4758 ms/op 2.2623 ms/op 1.09
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 31.814 ms/op 29.707 ms/op 1.07
BeaconState.hashTreeRoot - 1 balances 165.67 us/op 156.13 us/op 1.06
BeaconState.hashTreeRoot - 32 balances 1.4714 ms/op 1.3426 ms/op 1.10
BeaconState.hashTreeRoot - 512 balances 13.078 ms/op 11.820 ms/op 1.11
BeaconState.hashTreeRoot - 250000 balances 217.17 ms/op 225.34 ms/op 0.96
aggregationBits - 2048 els - zipIndexesInBitList 21.410 us/op 16.067 us/op 1.33
byteArrayEquals 32 83.007 ns/op 70.736 ns/op 1.17
Buffer.compare 32 61.084 ns/op 53.038 ns/op 1.15
byteArrayEquals 1024 2.2690 us/op 1.9314 us/op 1.17
Buffer.compare 1024 82.781 ns/op 68.500 ns/op 1.21
byteArrayEquals 16384 36.016 us/op 30.764 us/op 1.17
Buffer.compare 16384 287.93 ns/op 259.49 ns/op 1.11
byteArrayEquals 123687377 298.14 ms/op 232.04 ms/op 1.28
Buffer.compare 123687377 9.8371 ms/op 6.2015 ms/op 1.59
byteArrayEquals 32 - diff last byte 91.412 ns/op 71.372 ns/op 1.28
Buffer.compare 32 - diff last byte 64.319 ns/op 55.467 ns/op 1.16
byteArrayEquals 1024 - diff last byte 2.2147 us/op 2.0202 us/op 1.10
Buffer.compare 1024 - diff last byte 79.794 ns/op 71.718 ns/op 1.11
byteArrayEquals 16384 - diff last byte 36.330 us/op 32.211 us/op 1.13
Buffer.compare 16384 - diff last byte 299.12 ns/op 273.41 ns/op 1.09
byteArrayEquals 123687377 - diff last byte 269.97 ms/op 239.13 ms/op 1.13
Buffer.compare 123687377 - diff last byte 9.7981 ms/op 6.0965 ms/op 1.61
byteArrayEquals 32 - random bytes 8.0960 ns/op 5.1520 ns/op 1.57
Buffer.compare 32 - random bytes 70.138 ns/op 59.021 ns/op 1.19
byteArrayEquals 1024 - random bytes 6.7890 ns/op 5.0610 ns/op 1.34
Buffer.compare 1024 - random bytes 68.254 ns/op 58.590 ns/op 1.16
byteArrayEquals 16384 - random bytes 7.2930 ns/op 5.3160 ns/op 1.37
Buffer.compare 16384 - random bytes 68.257 ns/op 58.236 ns/op 1.17
byteArrayEquals 123687377 - random bytes 11.350 ns/op 8.3200 ns/op 1.36
Buffer.compare 123687377 - random bytes 82.470 ns/op 61.650 ns/op 1.34
regular array get 100000 times 51.206 us/op 43.111 us/op 1.19
wrappedArray get 100000 times 50.246 us/op 43.048 us/op 1.17
arrayWithProxy get 100000 times 15.635 ms/op 14.020 ms/op 1.12
ssz.Root.equals 63.641 ns/op 52.358 ns/op 1.22
byteArrayEquals 63.303 ns/op 51.512 ns/op 1.23
Buffer.compare 14.125 ns/op 10.542 ns/op 1.34
shuffle list - 16384 els 8.1725 ms/op 6.9002 ms/op 1.18
shuffle list - 250000 els 114.90 ms/op 99.823 ms/op 1.15
processSlot - 1 slots 21.978 us/op 16.517 us/op 1.33
processSlot - 32 slots 4.3655 ms/op 3.5401 ms/op 1.23
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 69.940 ms/op 60.992 ms/op 1.15
getCommitteeAssignments - req 1 vs - 250000 vc 3.1468 ms/op 2.4611 ms/op 1.28
getCommitteeAssignments - req 100 vs - 250000 vc 4.2271 ms/op 3.6420 ms/op 1.16
getCommitteeAssignments - req 1000 vs - 250000 vc 4.8759 ms/op 3.9976 ms/op 1.22
findModifiedValidators - 10000 modified validators 703.28 ms/op 559.22 ms/op 1.26
findModifiedValidators - 1000 modified validators 554.98 ms/op 425.51 ms/op 1.30
findModifiedValidators - 100 modified validators 525.34 ms/op 397.22 ms/op 1.32
findModifiedValidators - 10 modified validators 525.54 ms/op 372.77 ms/op 1.41
findModifiedValidators - 1 modified validators 514.01 ms/op 394.92 ms/op 1.30
findModifiedValidators - no difference 581.28 ms/op 403.22 ms/op 1.44
compare ViewDUs 5.5872 s/op 4.3589 s/op 1.28
compare each validator Uint8Array 2.2081 s/op 1.9616 s/op 1.13
compare ViewDU to Uint8Array 1.6922 s/op 1.1229 s/op 1.51
migrate state 1000000 validators, 24 modified, 0 new 936.40 ms/op 807.40 ms/op 1.16
migrate state 1000000 validators, 1700 modified, 1000 new 1.0868 s/op 1.1144 s/op 0.98
migrate state 1000000 validators, 3400 modified, 2000 new 1.4315 s/op 1.3544 s/op 1.06
migrate state 1500000 validators, 24 modified, 0 new 816.63 ms/op 818.38 ms/op 1.00
migrate state 1500000 validators, 1700 modified, 1000 new 1.1016 s/op 1.1023 s/op 1.00
migrate state 1500000 validators, 3400 modified, 2000 new 1.4866 s/op 1.3207 s/op 1.13
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.3800 ns/op 4.0900 ns/op 1.32
state getBlockRootAtSlot - 250000 vs - 7PWei 690.95 ns/op 706.77 ns/op 0.98
computeProposers - vc 250000 10.456 ms/op 9.2365 ms/op 1.13
computeEpochShuffling - vc 250000 111.71 ms/op 101.63 ms/op 1.10
getNextSyncCommittee - vc 250000 177.93 ms/op 154.61 ms/op 1.15
computeSigningRoot for AttestationData 27.559 us/op 27.064 us/op 1.02
hash AttestationData serialized data then Buffer.toString(base64) 2.5195 us/op 2.2756 us/op 1.11
toHexString serialized data 1.1867 us/op 1.0751 us/op 1.10
Buffer.toString(base64) 261.08 ns/op 217.78 ns/op 1.20

by benchmarkbot/action

nazarhussain
nazarhussain commented Jan 15, 2024
View reviewed changes
packages/beacon-node/test/spec/presets/fork_choice.test.ts
Comment on lines +115 to +116
// The handler of `ChainEvent.forkChoiceFinalized` access `db.block` and raise error if not found.
chain.emitter.removeAllListeners(ChainEvent.forkChoiceFinalized);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handler is throwing an error block not found, which is unhandled rejection even in previous implementation but never been caught by mocha.

nazarhussain
nazarhussain commented Jan 15, 2024
View reviewed changes
packages/validator/test/spec/spec.test.ts
Comment on lines +38 to +41
// If there is no `it` block then we should skip to avoid running `beforeEach` hooks
if (step.blocks.length === 0 && step.attestations.length === 0) {
continue;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very interesting case to show difference of both test runners.

  1. Mocha don't run the beforeEach which is defined within the loop without it.
  2. Vitest consider all it blocks and beforeEach blocks as children of the describe and run two beforeEach for single it block.

@wemeetagain wemeetagain merged commit 4996f4d into unstable Jan 15, 2024
12 of 15 checks passed
@wemeetagain wemeetagain deleted the nh/vitest-spec-test-util branch January 15, 2024 15:58
@nflaig nflaig mentioned this pull request Jan 16, 2024
Merged
ensi321 pushed a commit to ensi321/lodestar that referenced this pull request Jan 22, 2024
@nazarhussain @ensi321
test: migrate spec-test-utils and spec tests to vitest (ChainSafe#6222)
8d537f6 
* Migrate spec-test-utils and spec tests to vitest

* Fix build check

* Fix export path

* Add config file

* Fix spec tests

* Fix the duplicate package script

* Fix spec tests

* Fix lint errors

* Update the spec tests runner tasks

* Fix the import for a spec test

* Fix the import for a spec test

* Fix spec tests

* Update the task runner

* Run spec test in fork not threads

* Fix the config file path

* Fix validator spec tests

* Fix e2e tests

* Make the validator spec tests run in parallel

* Update the spec test runner options
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.15.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees

@nazarhussain nazarhussain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nazarhussain @wemeetagain