Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add POST endpoints for validators and validator_balances #6655

Merged
merged 2 commits into from
Apr 11, 2024
Merged

feat: add POST endpoints for validators and validator_balances #6655

merged 2 commits into from
Apr 11, 2024

Conversation

nflaig
Copy link
Member

@nflaig nflaig commented Apr 9, 2024

Motivation

Description

  • Adds POST endpoints for validators and validator_balances
  • More robust handling of numbers in getStateValidatorIndex

Closes #6058

Depends on #6645

@nflaig nflaig added this to the v1.18.0 milestone Apr 9, 2024
@codecov Codecov
Copy link

codecov bot commented Apr 9, 2024 •
edited
Loading

Codecov Report

Merging #6655 (3c9b04c) into unstable (5ccae1c) will increase coverage by 0.17%.
The diff coverage is 97.77%.

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #6655      +/-   ##
============================================
+ Coverage     61.64%   61.81%   +0.17%     
============================================
  Files           556      556              
  Lines         58782    59050     +268     
  Branches       1886     1898      +12     
============================================
+ Hits          36236    36502     +266     
  Misses        22505    22505              
- Partials         41       43       +2

Base automatically changed from nflaig/finalized-property to unstable April 11, 2024 10:39
@nflaig nflaig force-pushed the nflaig/post-endpoints branch 2 times, most recently from cdc92b2 to 39f226d Compare April 11, 2024 12:00
@nflaig nflaig force-pushed the nflaig/post-endpoints branch from 39f226d to a4c6048 Compare April 11, 2024 12:14
@nflaig nflaig marked this pull request as ready for review April 11, 2024 12:15
@nflaig nflaig requested a review from a team as a code owner April 11, 2024 12:15
jeluard
jeluard reviewed Apr 11, 2024
View reviewed changes
packages/api/src/beacon/routes/beacon/state.ts
}),
parseReq: ({params, body}) => [
params.state_id,
body?.map((id) => (typeof id === "string" && id.startsWith("0x") ? id : fromU64Str(id))),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

id is always a string? fromU64Str only accepts string.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, if we use string for validator index inside Lodestar it is likely unintended and should be fixed. And even though the type of fromU64Str states in only accepts strings (which imo it should), in practice it will handle numbers as type number just fine.

It only has to be a stringified number over the wire

apis/beacon/states/validators.yaml#L115-L120

ids:
 type: array
 uniqueItems: true
 items:
   description: "Either hex encoded public key (any bytes48 with 0x prefix) or validator index"
   type: string # <--

Copy link
Member Author

@nflaig nflaig Apr 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indices store strictly uses type number

lodestar/packages/validator/src/services/indices.ts

Lines 42 to 45 in 5ccae1c

export class IndicesService {
readonly index2pubkey = new Map<ValidatorIndex, PubkeyHex>();
/** Indexed by pubkey in hex 0x prefixed */
readonly pubkey2index = new Map<PubkeyHex, ValidatorIndex>();

packages/api/src/beacon/routes/beacon/state.ts Show resolved Hide resolved
packages/beacon-node/src/api/impl/beacon/state/utils.ts Outdated
}

if (typeof id === "number") {
validatorIndex = id;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having 2 declaration for validatorIndex with smaller scope would improve readability and prevent the issue you fixed.

Copy link
Member Author

@nflaig nflaig Apr 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue was really there because of a wrong type cast, so by removing that it should not be possible anymore. But your suggestion makes sense, simpler to follow state of validatorIndex, as it really doesn't need to be declared at the top of the function

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also added unit test to make sure we would catch any regression

packages/beacon-node/src/api/impl/beacon/state/utils.ts Outdated Show resolved Hide resolved
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 11, 2024 •
edited
Loading

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite Current: c2326f1 Previous: 5ccae1c Ratio
Map set x1000 18.222 ns/op 56.321 ns/op 0.32
Set add up to 64 items then delete first 1.6542 us/op 5.4720 us/op 0.30
Full benchmark results
Benchmark suite Current: c2326f1 Previous: 5ccae1c Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 378.13 us/op 842.34 us/op 0.45
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 51.803 us/op 81.774 us/op 0.63
BLS verify - blst-native 1.0704 ms/op 1.3026 ms/op 0.82
BLS verifyMultipleSignatures 3 - blst-native 2.2924 ms/op 2.7106 ms/op 0.85
BLS verifyMultipleSignatures 8 - blst-native 5.0912 ms/op 5.9283 ms/op 0.86
BLS verifyMultipleSignatures 32 - blst-native 18.320 ms/op 21.579 ms/op 0.85
BLS verifyMultipleSignatures 64 - blst-native 36.317 ms/op 42.952 ms/op 0.85
BLS verifyMultipleSignatures 128 - blst-native 72.126 ms/op 85.724 ms/op 0.84
BLS deserializing 10000 signatures 783.95 ms/op 871.17 ms/op 0.90
BLS deserializing 100000 signatures 8.1302 s/op 9.3481 s/op 0.87
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.1352 ms/op 1.2953 ms/op 0.88
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.2839 ms/op 1.4560 ms/op 0.88
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.0175 ms/op 2.7478 ms/op 0.73
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.7529 ms/op 4.2240 ms/op 0.89
BLS verifyMultipleSignatures - same message - 128 - blst-native 5.0109 ms/op 7.1888 ms/op 0.70
BLS aggregatePubkeys 32 - blst-native 22.350 us/op 26.605 us/op 0.84
BLS aggregatePubkeys 128 - blst-native 87.787 us/op 97.286 us/op 0.90
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 41.008 ms/op 58.582 ms/op 0.70
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 49.220 ms/op 61.278 ms/op 0.80
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 26.208 ms/op 35.832 ms/op 0.73
getSlashingsAndExits - default max 152.35 us/op 180.65 us/op 0.84
getSlashingsAndExits - 2k 642.85 us/op 400.11 us/op 1.61
proposeBlockBody type=full, size=empty 4.1989 ms/op 5.6710 ms/op 0.74
isKnown best case - 1 super set check 356.00 ns/op 320.00 ns/op 1.11
isKnown normal case - 2 super set checks 368.00 ns/op 321.00 ns/op 1.15
isKnown worse case - 16 super set checks 353.00 ns/op 310.00 ns/op 1.14
InMemoryCheckpointStateCache - add get delete 4.2370 us/op 5.9500 us/op 0.71
validate api signedAggregateAndProof - struct 2.3181 ms/op 2.7793 ms/op 0.83
validate gossip signedAggregateAndProof - struct 2.2987 ms/op 2.7545 ms/op 0.83
validate gossip attestation - vc 640000 1.1462 ms/op 1.3762 ms/op 0.83
batch validate gossip attestation - vc 640000 - chunk 32 147.58 us/op 187.24 us/op 0.79
batch validate gossip attestation - vc 640000 - chunk 64 132.41 us/op 174.01 us/op 0.76
batch validate gossip attestation - vc 640000 - chunk 128 147.30 us/op 167.44 us/op 0.88
batch validate gossip attestation - vc 640000 - chunk 256 114.22 us/op 152.05 us/op 0.75
pickEth1Vote - no votes 934.26 us/op 1.4720 ms/op 0.63
pickEth1Vote - max votes 5.5005 ms/op 11.710 ms/op 0.47
pickEth1Vote - Eth1Data hashTreeRoot value x2048 12.194 ms/op 23.748 ms/op 0.51
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 16.771 ms/op 38.931 ms/op 0.43
pickEth1Vote - Eth1Data fastSerialize value x2048 476.82 us/op 850.61 us/op 0.56
pickEth1Vote - Eth1Data fastSerialize tree x2048 5.9874 ms/op 7.0402 ms/op 0.85
bytes32 toHexString 473.00 ns/op 759.00 ns/op 0.62
bytes32 Buffer.toString(hex) 338.00 ns/op 367.00 ns/op 0.92
bytes32 Buffer.toString(hex) from Uint8Array 523.00 ns/op 596.00 ns/op 0.88
bytes32 Buffer.toString(hex) + 0x 346.00 ns/op 357.00 ns/op 0.97
Object access 1 prop 0.21200 ns/op 0.25700 ns/op 0.82
Map access 1 prop 0.19800 ns/op 0.16900 ns/op 1.17
Object get x1000 5.5520 ns/op 8.3220 ns/op 0.67
Map get x1000 0.88000 ns/op 0.90200 ns/op 0.98
Object set x1000 26.623 ns/op 71.964 ns/op 0.37
Map set x1000 18.222 ns/op 56.321 ns/op 0.32
Return object 10000 times 0.23830 ns/op 0.28320 ns/op 0.84
Throw Error 10000 times 2.8526 us/op 3.9249 us/op 0.73
fastMsgIdFn sha256 / 200 bytes 2.0980 us/op 3.5870 us/op 0.58
fastMsgIdFn h32 xxhash / 200 bytes 361.00 ns/op 376.00 ns/op 0.96
fastMsgIdFn h64 xxhash / 200 bytes 387.00 ns/op 454.00 ns/op 0.85
fastMsgIdFn sha256 / 1000 bytes 6.5120 us/op 11.980 us/op 0.54
fastMsgIdFn h32 xxhash / 1000 bytes 480.00 ns/op 507.00 ns/op 0.95
fastMsgIdFn h64 xxhash / 1000 bytes 488.00 ns/op 493.00 ns/op 0.99
fastMsgIdFn sha256 / 10000 bytes 54.630 us/op 109.46 us/op 0.50
fastMsgIdFn h32 xxhash / 10000 bytes 1.9100 us/op 2.1020 us/op 0.91
fastMsgIdFn h64 xxhash / 10000 bytes 1.3410 us/op 1.4420 us/op 0.93
send data - 1000 256B messages 14.537 ms/op 18.336 ms/op 0.79
send data - 1000 512B messages 18.267 ms/op 30.994 ms/op 0.59
send data - 1000 1024B messages 30.217 ms/op 41.166 ms/op 0.73
send data - 1000 1200B messages 32.895 ms/op 42.436 ms/op 0.78
send data - 1000 2048B messages 34.490 ms/op 54.628 ms/op 0.63
send data - 1000 4096B messages 36.527 ms/op 42.504 ms/op 0.86
send data - 1000 16384B messages 87.338 ms/op 121.58 ms/op 0.72
send data - 1000 65536B messages 398.74 ms/op 466.94 ms/op 0.85
enrSubnets - fastDeserialize 64 bits 1.0700 us/op 1.7220 us/op 0.62
enrSubnets - ssz BitVector 64 bits 447.00 ns/op 558.00 ns/op 0.80
enrSubnets - fastDeserialize 4 bits 224.00 ns/op 247.00 ns/op 0.91
enrSubnets - ssz BitVector 4 bits 492.00 ns/op 539.00 ns/op 0.91
prioritizePeers score -10:0 att 32-0.1 sync 2-0 152.30 us/op 243.18 us/op 0.63
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 200.29 us/op 323.13 us/op 0.62
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 344.02 us/op 398.86 us/op 0.86
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 388.41 us/op 657.41 us/op 0.59
prioritizePeers score 0:0 att 64-1 sync 4-1 509.61 us/op 843.98 us/op 0.60
array of 16000 items push then shift 1.3106 us/op 1.8282 us/op 0.72
LinkedList of 16000 items push then shift 7.3780 ns/op 10.022 ns/op 0.74
array of 16000 items push then pop 83.560 ns/op 112.32 ns/op 0.74
LinkedList of 16000 items push then pop 5.6730 ns/op 9.5540 ns/op 0.59
array of 24000 items push then shift 1.8929 us/op 2.7518 us/op 0.69
LinkedList of 24000 items push then shift 5.9040 ns/op 10.180 ns/op 0.58
array of 24000 items push then pop 92.235 ns/op 166.93 ns/op 0.55
LinkedList of 24000 items push then pop 5.8610 ns/op 9.7310 ns/op 0.60
intersect bitArray bitLen 8 4.9370 ns/op 6.1840 ns/op 0.80
intersect array and set length 8 58.567 ns/op 99.696 ns/op 0.59
intersect bitArray bitLen 128 29.249 ns/op 39.456 ns/op 0.74
intersect array and set length 128 757.08 ns/op 1.2213 us/op 0.62
bitArray.getTrueBitIndexes() bitLen 128 1.2190 us/op 1.9710 us/op 0.62
bitArray.getTrueBitIndexes() bitLen 248 2.0880 us/op 3.8000 us/op 0.55
bitArray.getTrueBitIndexes() bitLen 512 4.4310 us/op 7.7180 us/op 0.57
Buffer.concat 32 items 1.0940 us/op 1.3010 us/op 0.84
Uint8Array.set 32 items 1.7530 us/op 2.6220 us/op 0.67
Set add up to 64 items then delete first 1.6542 us/op 5.4720 us/op 0.30
OrderedSet add up to 64 items then delete first 2.5471 us/op 6.8881 us/op 0.37
Set add up to 64 items then delete last 1.9192 us/op 5.6614 us/op 0.34
OrderedSet add up to 64 items then delete last 2.8254 us/op 7.2747 us/op 0.39
Set add up to 64 items then delete middle 1.8697 us/op 5.4072 us/op 0.35
OrderedSet add up to 64 items then delete middle 3.9666 us/op 8.7194 us/op 0.45
Set add up to 128 items then delete first 3.8205 us/op 11.185 us/op 0.34
OrderedSet add up to 128 items then delete first 6.5096 us/op 15.514 us/op 0.42
Set add up to 128 items then delete last 3.7519 us/op 11.159 us/op 0.34
OrderedSet add up to 128 items then delete last 5.6837 us/op 14.508 us/op 0.39
Set add up to 128 items then delete middle 3.8194 us/op 10.948 us/op 0.35
OrderedSet add up to 128 items then delete middle 10.723 us/op 20.976 us/op 0.51
Set add up to 256 items then delete first 7.8230 us/op 22.988 us/op 0.34
OrderedSet add up to 256 items then delete first 12.273 us/op 31.916 us/op 0.38
Set add up to 256 items then delete last 7.5877 us/op 22.172 us/op 0.34
OrderedSet add up to 256 items then delete last 11.491 us/op 29.506 us/op 0.39
Set add up to 256 items then delete middle 7.6875 us/op 22.327 us/op 0.34
OrderedSet add up to 256 items then delete middle 34.834 us/op 53.999 us/op 0.65
transfer serialized Status (84 B) 1.7520 us/op 2.1080 us/op 0.83
copy serialized Status (84 B) 1.3420 us/op 1.5530 us/op 0.86
transfer serialized SignedVoluntaryExit (112 B) 1.7350 us/op 2.1350 us/op 0.81
copy serialized SignedVoluntaryExit (112 B) 1.5410 us/op 1.5450 us/op 1.00
transfer serialized ProposerSlashing (416 B) 3.3080 us/op 2.4290 us/op 1.36
copy serialized ProposerSlashing (416 B) 2.3650 us/op 2.1530 us/op 1.10
transfer serialized Attestation (485 B) 2.2410 us/op 2.4980 us/op 0.90
copy serialized Attestation (485 B) 1.8490 us/op 2.1900 us/op 0.84
transfer serialized AttesterSlashing (33232 B) 2.5530 us/op 2.2110 us/op 1.15
copy serialized AttesterSlashing (33232 B) 6.4610 us/op 12.617 us/op 0.51
transfer serialized Small SignedBeaconBlock (128000 B) 2.4860 us/op 4.1320 us/op 0.60
copy serialized Small SignedBeaconBlock (128000 B) 18.824 us/op 29.661 us/op 0.63
transfer serialized Avg SignedBeaconBlock (200000 B) 2.9020 us/op 5.4710 us/op 0.53
copy serialized Avg SignedBeaconBlock (200000 B) 33.839 us/op 47.753 us/op 0.71
transfer serialized BlobsSidecar (524380 B) 4.8820 us/op 5.9490 us/op 0.82
copy serialized BlobsSidecar (524380 B) 79.920 us/op 146.00 us/op 0.55
transfer serialized Big SignedBeaconBlock (1000000 B) 3.8270 us/op 6.0870 us/op 0.63
copy serialized Big SignedBeaconBlock (1000000 B) 264.38 us/op 302.84 us/op 0.87
pass gossip attestations to forkchoice per slot 2.8427 ms/op 5.0695 ms/op 0.56
forkChoice updateHead vc 100000 bc 64 eq 0 471.00 us/op 762.02 us/op 0.62
forkChoice updateHead vc 600000 bc 64 eq 0 4.7796 ms/op 4.6831 ms/op 1.02
forkChoice updateHead vc 1000000 bc 64 eq 0 5.0531 ms/op 9.1242 ms/op 0.55
forkChoice updateHead vc 600000 bc 320 eq 0 3.2300 ms/op 5.0409 ms/op 0.64
forkChoice updateHead vc 600000 bc 1200 eq 0 2.8221 ms/op 5.0094 ms/op 0.56
forkChoice updateHead vc 600000 bc 7200 eq 0 3.8812 ms/op 6.2432 ms/op 0.62
forkChoice updateHead vc 600000 bc 64 eq 1000 10.185 ms/op 13.251 ms/op 0.77
forkChoice updateHead vc 600000 bc 64 eq 10000 10.055 ms/op 14.724 ms/op 0.68
forkChoice updateHead vc 600000 bc 64 eq 300000 12.861 ms/op 27.688 ms/op 0.46
computeDeltas 500000 validators 300 proto nodes 3.8752 ms/op 7.6893 ms/op 0.50
computeDeltas 500000 validators 1200 proto nodes 3.1629 ms/op 7.8282 ms/op 0.40
computeDeltas 500000 validators 7200 proto nodes 3.0075 ms/op 7.2236 ms/op 0.42
computeDeltas 750000 validators 300 proto nodes 4.9086 ms/op 11.051 ms/op 0.44
computeDeltas 750000 validators 1200 proto nodes 4.7626 ms/op 11.550 ms/op 0.41
computeDeltas 750000 validators 7200 proto nodes 4.5988 ms/op 11.073 ms/op 0.42
computeDeltas 1400000 validators 300 proto nodes 9.2740 ms/op 21.337 ms/op 0.43
computeDeltas 1400000 validators 1200 proto nodes 9.6491 ms/op 19.634 ms/op 0.49
computeDeltas 1400000 validators 7200 proto nodes 10.367 ms/op 20.173 ms/op 0.51
computeDeltas 2100000 validators 300 proto nodes 14.467 ms/op 29.856 ms/op 0.48
computeDeltas 2100000 validators 1200 proto nodes 13.791 ms/op 28.879 ms/op 0.48
computeDeltas 2100000 validators 7200 proto nodes 13.287 ms/op 28.634 ms/op 0.46
altair processAttestation - 250000 vs - 7PWei normalcase 2.5502 ms/op 2.3347 ms/op 1.09
altair processAttestation - 250000 vs - 7PWei worstcase 2.7282 ms/op 3.4971 ms/op 0.78
altair processAttestation - setStatus - 1/6 committees join 111.28 us/op 157.69 us/op 0.71
altair processAttestation - setStatus - 1/3 committees join 209.66 us/op 301.08 us/op 0.70
altair processAttestation - setStatus - 1/2 committees join 348.31 us/op 412.85 us/op 0.84
altair processAttestation - setStatus - 2/3 committees join 417.78 us/op 501.87 us/op 0.83
altair processAttestation - setStatus - 4/5 committees join 552.70 us/op 701.71 us/op 0.79
altair processAttestation - setStatus - 100% committees join 631.73 us/op 826.99 us/op 0.76
altair processBlock - 250000 vs - 7PWei normalcase 8.5836 ms/op 8.6688 ms/op 0.99
altair processBlock - 250000 vs - 7PWei normalcase hashState 33.561 ms/op 37.142 ms/op 0.90
altair processBlock - 250000 vs - 7PWei worstcase 36.750 ms/op 46.414 ms/op 0.79
altair processBlock - 250000 vs - 7PWei worstcase hashState 91.661 ms/op 92.480 ms/op 0.99
phase0 processBlock - 250000 vs - 7PWei normalcase 2.4107 ms/op 2.7049 ms/op 0.89
phase0 processBlock - 250000 vs - 7PWei worstcase 28.682 ms/op 32.013 ms/op 0.90
altair processEth1Data - 250000 vs - 7PWei normalcase 337.49 us/op 660.31 us/op 0.51
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 14.089 us/op 9.5110 us/op 1.48
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 73.624 us/op 71.451 us/op 1.03
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 23.338 us/op 14.849 us/op 1.57
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 15.654 us/op 17.387 us/op 0.90
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 163.51 us/op 192.28 us/op 0.85
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.0639 ms/op 1.1438 ms/op 0.93
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.7892 ms/op 1.6277 ms/op 1.10
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.4932 ms/op 1.5716 ms/op 0.95
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 3.9882 ms/op 3.8604 ms/op 1.03
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.2763 ms/op 2.3468 ms/op 0.97
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 5.7027 ms/op 4.6973 ms/op 1.21
Tree 40 250000 create 426.41 ms/op 324.36 ms/op 1.31
Tree 40 250000 get(125000) 160.81 ns/op 194.04 ns/op 0.83
Tree 40 250000 set(125000) 796.22 ns/op 977.17 ns/op 0.81
Tree 40 250000 toArray() 28.157 ms/op 20.080 ms/op 1.40
Tree 40 250000 iterate all - toArray() + loop 24.658 ms/op 20.092 ms/op 1.23
Tree 40 250000 iterate all - get(i) 56.105 ms/op 64.712 ms/op 0.87
MutableVector 250000 create 14.012 ms/op 11.500 ms/op 1.22
MutableVector 250000 get(125000) 5.9360 ns/op 6.4410 ns/op 0.92
MutableVector 250000 set(125000) 485.41 ns/op 279.93 ns/op 1.73
MutableVector 250000 toArray() 3.9889 ms/op 3.0997 ms/op 1.29
MutableVector 250000 iterate all - toArray() + loop 3.9004 ms/op 3.5426 ms/op 1.10
MutableVector 250000 iterate all - get(i) 1.4227 ms/op 1.5274 ms/op 0.93
Array 250000 create 3.1061 ms/op 2.8344 ms/op 1.10
Array 250000 clone - spread 1.3522 ms/op 1.1945 ms/op 1.13
Array 250000 get(125000) 1.1300 ns/op 1.0080 ns/op 1.12
Array 250000 set(125000) 1.3180 ns/op 4.1140 ns/op 0.32
Array 250000 iterate all - loop 160.43 us/op 163.29 us/op 0.98
effectiveBalanceIncrements clone Uint8Array 300000 15.524 us/op 26.781 us/op 0.58
effectiveBalanceIncrements clone MutableVector 300000 448.00 ns/op 340.00 ns/op 1.32
effectiveBalanceIncrements rw all Uint8Array 300000 192.18 us/op 195.78 us/op 0.98
effectiveBalanceIncrements rw all MutableVector 300000 69.831 ms/op 78.718 ms/op 0.89
phase0 afterProcessEpoch - 250000 vs - 7PWei 79.638 ms/op 108.65 ms/op 0.73
phase0 beforeProcessEpoch - 250000 vs - 7PWei 48.147 ms/op 52.053 ms/op 0.92
altair processEpoch - mainnet_e81889 412.16 ms/op 486.59 ms/op 0.85
mainnet_e81889 - altair beforeProcessEpoch 90.464 ms/op 82.312 ms/op 1.10
mainnet_e81889 - altair processJustificationAndFinalization 24.685 us/op 14.417 us/op 1.71
mainnet_e81889 - altair processInactivityUpdates 6.7406 ms/op 6.4702 ms/op 1.04
mainnet_e81889 - altair processRewardsAndPenalties 78.060 ms/op 69.910 ms/op 1.12
mainnet_e81889 - altair processRegistryUpdates 3.8760 us/op 2.3840 us/op 1.63
mainnet_e81889 - altair processSlashings 702.00 ns/op 464.00 ns/op 1.51
mainnet_e81889 - altair processEth1DataReset 889.00 ns/op 504.00 ns/op 1.76
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.0504 ms/op 1.3315 ms/op 0.79
mainnet_e81889 - altair processSlashingsReset 3.8610 us/op 4.2100 us/op 0.92
mainnet_e81889 - altair processRandaoMixesReset 6.6700 us/op 4.9590 us/op 1.35
mainnet_e81889 - altair processHistoricalRootsUpdate 874.00 ns/op 823.00 ns/op 1.06
mainnet_e81889 - altair processParticipationFlagUpdates 2.2000 us/op 1.5550 us/op 1.41
mainnet_e81889 - altair processSyncCommitteeUpdates 799.00 ns/op 783.00 ns/op 1.02
mainnet_e81889 - altair afterProcessEpoch 81.420 ms/op 117.95 ms/op 0.69
capella processEpoch - mainnet_e217614 1.6837 s/op 2.0011 s/op 0.84
mainnet_e217614 - capella beforeProcessEpoch 413.20 ms/op 458.64 ms/op 0.90
mainnet_e217614 - capella processJustificationAndFinalization 12.746 us/op 13.732 us/op 0.93
mainnet_e217614 - capella processInactivityUpdates 14.250 ms/op 20.368 ms/op 0.70
mainnet_e217614 - capella processRewardsAndPenalties 401.50 ms/op 504.54 ms/op 0.80
mainnet_e217614 - capella processRegistryUpdates 19.366 us/op 16.390 us/op 1.18
mainnet_e217614 - capella processSlashings 714.00 ns/op 537.00 ns/op 1.33
mainnet_e217614 - capella processEth1DataReset 371.00 ns/op 420.00 ns/op 0.88
mainnet_e217614 - capella processEffectiveBalanceUpdates 3.1338 ms/op 4.3413 ms/op 0.72
mainnet_e217614 - capella processSlashingsReset 2.6160 us/op 2.6920 us/op 0.97
mainnet_e217614 - capella processRandaoMixesReset 3.1970 us/op 4.4290 us/op 0.72
mainnet_e217614 - capella processHistoricalRootsUpdate 692.00 ns/op 652.00 ns/op 1.06
mainnet_e217614 - capella processParticipationFlagUpdates 2.1070 us/op 1.9030 us/op 1.11
mainnet_e217614 - capella afterProcessEpoch 218.06 ms/op 350.26 ms/op 0.62
phase0 processEpoch - mainnet_e58758 374.36 ms/op 467.15 ms/op 0.80
mainnet_e58758 - phase0 beforeProcessEpoch 123.07 ms/op 116.85 ms/op 1.05
mainnet_e58758 - phase0 processJustificationAndFinalization 16.787 us/op 15.564 us/op 1.08
mainnet_e58758 - phase0 processRewardsAndPenalties 47.816 ms/op 51.794 ms/op 0.92
mainnet_e58758 - phase0 processRegistryUpdates 8.6150 us/op 9.3610 us/op 0.92
mainnet_e58758 - phase0 processSlashings 665.00 ns/op 560.00 ns/op 1.19
mainnet_e58758 - phase0 processEth1DataReset 579.00 ns/op 420.00 ns/op 1.38
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 784.20 us/op 1.1011 ms/op 0.71
mainnet_e58758 - phase0 processSlashingsReset 2.7560 us/op 2.8040 us/op 0.98
mainnet_e58758 - phase0 processRandaoMixesReset 3.2310 us/op 3.4460 us/op 0.94
mainnet_e58758 - phase0 processHistoricalRootsUpdate 586.00 ns/op 410.00 ns/op 1.43
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.0350 us/op 3.8420 us/op 0.79
mainnet_e58758 - phase0 afterProcessEpoch 68.064 ms/op 107.78 ms/op 0.63
phase0 processEffectiveBalanceUpdates - 250000 normalcase 965.97 us/op 1.3109 ms/op 0.74
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.0740 ms/op 1.4417 ms/op 0.74
altair processInactivityUpdates - 250000 normalcase 28.354 ms/op 22.857 ms/op 1.24
altair processInactivityUpdates - 250000 worstcase 25.549 ms/op 22.820 ms/op 1.12
phase0 processRegistryUpdates - 250000 normalcase 6.1690 us/op 8.9190 us/op 0.69
phase0 processRegistryUpdates - 250000 badcase_full_deposits 449.81 us/op 384.92 us/op 1.17
phase0 processRegistryUpdates - 250000 worstcase 0.5 124.60 ms/op 128.64 ms/op 0.97
altair processRewardsAndPenalties - 250000 normalcase 61.411 ms/op 51.648 ms/op 1.19
altair processRewardsAndPenalties - 250000 worstcase 73.153 ms/op 66.740 ms/op 1.10
phase0 getAttestationDeltas - 250000 normalcase 5.6384 ms/op 9.1945 ms/op 0.61
phase0 getAttestationDeltas - 250000 worstcase 6.8999 ms/op 9.2697 ms/op 0.74
phase0 processSlashings - 250000 worstcase 94.356 us/op 105.61 us/op 0.89
altair processSyncCommitteeUpdates - 250000 110.57 ms/op 154.36 ms/op 0.72
BeaconState.hashTreeRoot - No change 429.00 ns/op 399.00 ns/op 1.08
BeaconState.hashTreeRoot - 1 full validator 113.53 us/op 138.40 us/op 0.82
BeaconState.hashTreeRoot - 32 full validator 1.5690 ms/op 1.3144 ms/op 1.19
BeaconState.hashTreeRoot - 512 full validator 17.665 ms/op 16.212 ms/op 1.09
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 106.24 us/op 179.30 us/op 0.59
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.6352 ms/op 1.9074 ms/op 0.86
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 18.197 ms/op 23.656 ms/op 0.77
BeaconState.hashTreeRoot - 1 balances 83.523 us/op 107.32 us/op 0.78
BeaconState.hashTreeRoot - 32 balances 728.79 us/op 970.29 us/op 0.75
BeaconState.hashTreeRoot - 512 balances 7.2620 ms/op 10.646 ms/op 0.68
BeaconState.hashTreeRoot - 250000 balances 167.97 ms/op 184.18 ms/op 0.91
aggregationBits - 2048 els - zipIndexesInBitList 19.406 us/op 30.086 us/op 0.65
byteArrayEquals 32 65.189 ns/op 76.798 ns/op 0.85
Buffer.compare 32 38.955 ns/op 56.469 ns/op 0.69
byteArrayEquals 1024 1.7732 us/op 2.0557 us/op 0.86
Buffer.compare 1024 46.222 ns/op 71.198 ns/op 0.65
byteArrayEquals 16384 28.527 us/op 32.955 us/op 0.87
Buffer.compare 16384 227.47 ns/op 259.51 ns/op 0.88
byteArrayEquals 123687377 212.96 ms/op 247.05 ms/op 0.86
Buffer.compare 123687377 6.5496 ms/op 6.6102 ms/op 0.99
byteArrayEquals 32 - diff last byte 62.755 ns/op 74.718 ns/op 0.84
Buffer.compare 32 - diff last byte 40.388 ns/op 56.348 ns/op 0.72
byteArrayEquals 1024 - diff last byte 1.7645 us/op 2.0388 us/op 0.87
Buffer.compare 1024 - diff last byte 46.120 ns/op 71.683 ns/op 0.64
byteArrayEquals 16384 - diff last byte 28.116 us/op 32.360 us/op 0.87
Buffer.compare 16384 - diff last byte 231.18 ns/op 257.65 ns/op 0.90
byteArrayEquals 123687377 - diff last byte 212.76 ms/op 246.28 ms/op 0.86
Buffer.compare 123687377 - diff last byte 4.9051 ms/op 8.0701 ms/op 0.61
byteArrayEquals 32 - random bytes 4.6130 ns/op 5.3660 ns/op 0.86
Buffer.compare 32 - random bytes 40.880 ns/op 61.907 ns/op 0.66
byteArrayEquals 1024 - random bytes 4.5020 ns/op 5.2440 ns/op 0.86
Buffer.compare 1024 - random bytes 39.732 ns/op 60.847 ns/op 0.65
byteArrayEquals 16384 - random bytes 4.5090 ns/op 5.2600 ns/op 0.86
Buffer.compare 16384 - random bytes 40.765 ns/op 66.902 ns/op 0.61
byteArrayEquals 123687377 - random bytes 8.0500 ns/op 8.9600 ns/op 0.90
Buffer.compare 123687377 - random bytes 42.930 ns/op 71.350 ns/op 0.60
regular array get 100000 times 42.849 us/op 45.441 us/op 0.94
wrappedArray get 100000 times 42.734 us/op 45.223 us/op 0.94
arrayWithProxy get 100000 times 10.263 ms/op 15.437 ms/op 0.66
ssz.Root.equals 57.655 ns/op 55.952 ns/op 1.03
byteArrayEquals 55.256 ns/op 53.623 ns/op 1.03
Buffer.compare 9.7110 ns/op 11.791 ns/op 0.82
shuffle list - 16384 els 5.5680 ms/op 8.9517 ms/op 0.62
shuffle list - 250000 els 80.900 ms/op 128.81 ms/op 0.63
processSlot - 1 slots 15.995 us/op 15.343 us/op 1.04
processSlot - 32 slots 3.7133 ms/op 3.0698 ms/op 1.21
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 50.619 ms/op 55.973 ms/op 0.90
getCommitteeAssignments - req 1 vs - 250000 vc 2.4277 ms/op 2.6760 ms/op 0.91
getCommitteeAssignments - req 100 vs - 250000 vc 3.6831 ms/op 3.8599 ms/op 0.95
getCommitteeAssignments - req 1000 vs - 250000 vc 3.9698 ms/op 4.2532 ms/op 0.93
findModifiedValidators - 10000 modified validators 307.73 ms/op 381.24 ms/op 0.81
findModifiedValidators - 1000 modified validators 175.81 ms/op 206.55 ms/op 0.85
findModifiedValidators - 100 modified validators 167.88 ms/op 179.86 ms/op 0.93
findModifiedValidators - 10 modified validators 159.26 ms/op 191.25 ms/op 0.83
findModifiedValidators - 1 modified validators 165.92 ms/op 190.57 ms/op 0.87
findModifiedValidators - no difference 200.57 ms/op 210.36 ms/op 0.95
compare ViewDUs 4.2470 s/op 4.6739 s/op 0.91
compare each validator Uint8Array 1.2499 s/op 1.7027 s/op 0.73
compare ViewDU to Uint8Array 1.1886 s/op 1.2766 s/op 0.93
migrate state 1000000 validators, 24 modified, 0 new 738.93 ms/op 762.30 ms/op 0.97
migrate state 1000000 validators, 1700 modified, 1000 new 1.0371 s/op 1.0501 s/op 0.99
migrate state 1000000 validators, 3400 modified, 2000 new 1.2770 s/op 1.3126 s/op 0.97
migrate state 1500000 validators, 24 modified, 0 new 749.95 ms/op 813.79 ms/op 0.92
migrate state 1500000 validators, 1700 modified, 1000 new 974.17 ms/op 1.0772 s/op 0.90
migrate state 1500000 validators, 3400 modified, 2000 new 1.2715 s/op 1.3023 s/op 0.98
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.1700 ns/op 5.0100 ns/op 1.03
state getBlockRootAtSlot - 250000 vs - 7PWei 886.49 ns/op 918.29 ns/op 0.97
computeProposers - vc 250000 9.3215 ms/op 10.276 ms/op 0.91
computeEpochShuffling - vc 250000 92.849 ms/op 131.98 ms/op 0.70
getNextSyncCommittee - vc 250000 110.59 ms/op 174.98 ms/op 0.63
computeSigningRoot for AttestationData 25.513 us/op 27.442 us/op 0.93
hash AttestationData serialized data then Buffer.toString(base64) 1.3925 us/op 2.4209 us/op 0.58
toHexString serialized data 832.53 ns/op 1.1132 us/op 0.75
Buffer.toString(base64) 176.07 ns/op 216.90 ns/op 0.81

by benchmarkbot/action

@jeluard jeluard self-requested a review April 11, 2024 16:10
@nflaig nflaig merged commit 669239b into unstable Apr 11, 2024
19 of 20 checks passed
@nflaig nflaig deleted the nflaig/post-endpoints branch April 11, 2024 17:41
@nflaig nflaig mentioned this pull request Apr 12, 2024
Closed
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.18.0 🎉

nflaig added a commit that referenced this pull request Apr 28, 2024
@nflaig
Apply changes from #6655
8218615
wemeetagain added a commit that referenced this pull request Jun 10, 2024
@nflaig @wemeetagain
feat: provide first-class ssz support on api (#6749)
f8593a9 
* Add config route definitions

* Add debug route definitions

* Add events route description

* Add lightclient route definitions

* Flatten function params

* Type safety for optional params in write / parse req

* Method args are optional if only optional props

* Fix genesisValidatorsRoot type issue

* Revert requiring all params in write / parse req

* Update http client errors

* Add lodestar route definitions

* Add node route definitions

* Add proof route definitions

* Add builder route definitions

* Add validator route definitions

* Application method response can be void

* Generic options can be passed to application methods

* Default endpoint request type has body property

* Improve types of transform methods

* Export server types from index (to be removed)

* Update config api impl

* Update lightclient api impl

* Update events api impl

* Update lodestar api impl

* Update proof api impl

* Update node api impl

* Update debug api impl

* Update state api impl

* Update pool api impl

* Update blocks api impl

* Partially update validator api impl

* Update beacon routes export

* Align submitPoolBlsToExecutionChange method args

* Filters are always a object

* Update errors messages

* Add beacon client methods

* Add missing routeId label to stream time metric

* Fix json casing in codecs

* Apply remaining changes from #6227

* Produce block apis only have version meta

* Add block values meta to all produce block apis

* Apply changes from #6337

* Handle unsafe version in WithMeta and WithVersion

* Restore server api error

* Update fastify route types

* Update server routes / handlers

* Remove unnecessary type cast

* Restore per route clients

* Fix beacon route types

* Remove option to patch fetch from http client

* Update eventstream client, remove fetch override

Fallback does not work like this, see #6180 for proper solution

* Use StringType for validator status until #6059

* Remove empty fetch.ts file

* Add a few todos

* Update builder client and server routes

* Update beacon exports

* Update api index exports

* Update builder index imports

* Improve type safety of schema definitions

* Add headers to fastify schema

* Fix schema definition type

* Add missing schemas to route definitions

* Fix response codec type

* Remove response codec type casts

* Fix casing in json only codec

* Reuse EmptyResponseCodec

* Update base rest api server

* Update keymanager routes, client and server

* Reuse data types in keymanager impl

* Do not await setting headers, not a promise

* Improve type safety of empty codecs

* Only require to implement supported req methods

* Handle requests that only support one format

* Handle responses that only support one format

* Add json / ssz only req codecs

* Update only support errors

* Fix assertion

* Set correct accept header if only supports one format

* Fix eslint / prettier issues

* More formatting fixes

* Fix fallback request retries in case of http errors

* Formatting of res.error

* Add add retry functionality to http client (from #6387)

* Update rewards routes and server (#6178 and #6260)

* Allow to omit body in ssz req if not defined

* Always set metadata headers in response

* Cache wire format in api response

* Only call raw body for json meta

* Update api package tests (wip)

* Test json and ssz format in generic server tests

* Add a bunch of todos

* Fix a few broken route definitions

* Fix partial config test

* Another todo note

* Stringify body of json requests

* Override default response json method

* Validate external consensus version headers in request

* Add error handling todo

* Skip body schema validation for ssz request bodies

* Clean up generic server tests

* Pass node, proof, events generic tests

* Use enum for media types

* Fix a bunch of route definitions

* Add justified to blockid type

* Properly handle booleans, remove block values codec

* Create Uint8Array test data without allocating Buffer

* Let fastify handle Buffer conversion

* Convert Buffer to Uint8Array in content type parser

* Fix build issues

* Fix fork type in builder routes

* Add some notes

* Properly parse request headers

* Fix incorrect type assumptions in transform

* Generic server tests are passing (except lightclient)

* Correctly handle APIs with empty responses

* Update getHeader return type to reflect no bid responses

* Do not append '?' to URL if query string is empty

* Let server handler set status code for parsing errors

* Remove unused import

* Rename function, request specific

* Completely drop ssz support from getSpec

* Spec tests are passing against latest releases

* Drop unused fastify route config

* Drop ssz request from builder routes, not yet supported

* Remove import

* Apply change from #6695

* Update execution optimistic meta

* Apply changes from #6645

* Add workaround to fix epoch committees type issue

* Add todo to fix inefficient state conversion

* Convert committee to normal array

* Apply changes from #6655

* Align args of validators endpoints

* Convert indices to str in rewards apis

* Update api spec version of README badges

* Revert table formatting changes

* Make this accessible for class-basd API implementations

* Throw err if metadata is accessed for failed response

* Add assertOk to api response

* Tweak api error message

* Update operationIds match spec value

* Add missing version to blob sidecars metadata

* Test headers and ssz bodies against spec

* Minor reordering of code in spec parsing

* submitBlindedBlock throws err if fork is not execution

* responseOk might be undefined

* Remove statusOk from route definition

* Remove stale comment

* Less build errors in beacon-node

* getBlobSidecars return version from server impl

* Update validator produce block impl

* More expressive pool method args

* Application methods might be undefined in mock implementations

* Adress open TODOs in server handler

* Api response methods are synchronous now

* Fix all remaining build issues

* Use more performant from/toHex in server api impls

* Clean up some TODOs

* Fix ApiError type

* Errors related to parsing return a 400 status code

* Simplify method binding

* Forward api context to application methods

* There is no easy way to make generic opts typesafe

* Better separation of server / client code

* Fix comment about missing builder bid

* Remove todo, not worth the change / extra indentation

* Rename route definitions functions

* Return 400 if data passed to keymanager is invalid

* Properly handle response metadata headers

* Fix lint issues

* Add header jsdoc

* Move metadata related code into separate file

* Remove ssz from POST requests without body

* Only set content-type header if body exists

* Fix headers extra

* POST requests without body are handled similar to GET requests

* Fix http client options tests

* Improve validation and type safety of JSON metadata

* Add type guard for request without body

* Differentiate based on body instead of GET vs POST

* More renaming

* Simplify RequestCode type

* Review routes, improve validation

* Remaining local diff

* Fix accept header handling if only support one wire format

* Update 406 error to more closely match spec example

* Enforce version header via custom check instead of schema

* Use ssz as default request wire format

* Log failure to receive head event to verbose

* Do not set default value for context

* Update getClient return type to better align with method name

* Consistent pattern to get route definitions

* Dedupe api client type for builder and keymanager

* Fix fallback logic if server returns http error

* Update head event error logging

* Retry 415 errors with JSON and cache SSZ not supported

* Use fetch spy to assert call times

* Update comment

* Update getLightClientUpdatesByRange endpoint meta

* Do not forward ssz bytes of blinded block to publishBlock

* Fix lightclient e2e tests

* Version header in publishBlock api is optional

* Reduce type duplication

* Add option to override request init in route definition

* Add JsonOnlyResp codec

* Validate boolean str value from headers

* Document default wire formats

* Simplify merging of inits in http client

* Remove type hacks from fetchBeaconHealth

* Reduce call stack in http client

* Add .ssz() equivalent method for json to api response

* More http client tests

* Ensure topics query is provided to eventstream api

* Validate request content type in handler

Fastify does not cover all edge cases

* Review routes, fix param docs, no empty comments

* Fix typo

* Add note about builder spec not supporting ssz

* Consistently move keymanager jsdoc to routes

* Sanitize user provided init values before merging

* Remove unused ssz only codec

* Allow passing wire formats as string literals

* chore: review proof routes (#6843)

Review proof routes

* chore: review lightclient routes (#6842)

Review lightclient routes

* chore: review node routes (#6844)

Review node routes

* feat: add cli flags to configure http wire format (#6840)

* Review PR, mostly cosmetic changes

* Fix event stream error handling

---------

Co-authored-by: Cayman <caymannava@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jeluard jeluard jeluard approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.18.0
Development

Successfully merging this pull request may close these issues.

Implement POST endpoints for validators and validator_balances
3 participants
@nflaig @wemeetagain @jeluard