rpc v2: backpressure chainHead_v1_storage #5741

niklasad1 · 2024-09-17T14:43:04Z

This PR makes it possible for rpc_v2::Storage::query_iter_paginated to be "backpressured" which is achieved by having a channel where the result is sent back and when this channel is "full" we pause the iteration.

The chainHead_follow has an internal channel which doesn't represent the actual connection and that is set to a very small number (16). Recall that the JSON-RPC server has a dedicate buffer for each connection by default of 64.

Benchmarks using subxt on localhost

Iterate over 10 accounts on westend-dev -> ~2-3x faster
Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster
Fetch 1024 descendant values -> ~500x faster

The reason for this is because as Josep explained in the issue is that one is only allowed query five storage items per call and clients has make lots of calls to drive it forward..

substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs

substrate/client/rpc-spec-v2/src/chain_head/tests.rs

…iter

paritytech-cicd-pr · 2024-09-17T15:48:16Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 2/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7365923

…iter

substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs

niklasad1 · 2024-09-19T09:38:56Z

substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs

+impl OperationState {
+	pub fn stop(&mut self) {
+		if !self.stop.is_stopped() {
+			self.operations.lock().remove(&self.operation_id);


annoying to lock the mutex here instead of AtomicBool but this is needed to have an async notification when operation is stopped.

couldn't find a better way for this

I think this is acceptable since the operationStop shouldn't be happening too often if at all. We are also acquiring the mutex on dropping RegisteredOperations to clean up the tracing of operation IDs.

substrate/client/rpc-spec-v2/src/chain_head/chain_head.rs

substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs

lexnv

LGTM! Thanks for tackling this 🙏

Tiny nits around the return value of the chainHead_continue method and some open question about the number of reserved operation that should discard items

Co-authored-by: James Wilson <james@jsdw.me>

jsdw

LGTM and a nice general improvement in the code!

…iter

carlosala · 2024-10-17T14:15:54Z

Hey! Just circling back on this, do you think it is feasible to backport this PR into stable2407 and stable2409. It is actually a huge perf improvement.
Thanks!

cc @niklasad1 @jsdw @lexnv

Close #5589 This PR makes it possible for `rpc_v2::Storage::query_iter_paginated` to be "backpressured" which is achieved by having a channel where the result is sent back and when this channel is "full" we pause the iteration. The chainHead_follow has an internal channel which doesn't represent the actual connection and that is set to a very small number (16). Recall that the JSON-RPC server has a dedicate buffer for each connection by default of 64. - Because `archive_storage` also depends on `rpc_v2::Storage::query_iter_paginated` I had to tweak the method to support limits as well. The reason is that archive_storage won't get backpressured properly because it's not an subscription. (it would much easier if it would be a subscription in rpc v2 spec because nothing against querying huge amount storage keys) - `query_iter_paginated` doesn't necessarily return the storage "in order" such as - `query_iter_paginated(vec![("key1", hash), ("key2", value)], ...)` could return them in arbitrary order because it's wrapped in FuturesUnordered but I could change that if we want to process it inorder (it's slower) - there is technically no limit on the number of storage queries in each `chainHead_v1_storage call` rather than the rpc max message limit which 10MB and only allowed to max 16 calls `chainHead_v1_x` concurrently (this should be fine) - Iterate over 10 accounts on westend-dev -> ~2-3x faster - Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster - Fetch 1024 descendant values -> ~500x faster The reason for this is because as Josep explained in the issue is that one is only allowed query five storage items per call and clients has make lots of calls to drive it forward.. --------- Co-authored-by: command-bot <> Co-authored-by: James Wilson <james@jsdw.me>

niklasad1 · 2024-10-17T15:05:13Z

Hey! Just circling back on this, do you think it is feasible to backport this PR into stable2407 and stable2409. It is actually a huge perf improvement.
Thanks!

I had a look and stable2409 looks possible to backport and I have created a PR for it but stable2407 requires a bunch of others PRs that are not backported. Hopefully stable2409 is sufficient?

carlosala · 2024-10-18T07:48:35Z

I'd be great to backport all recent PRs around rpc server v2 to both supported versions. This will ensure that nodes get those fixes faster. I leave it to you to decide 👍🏻

Close #5589 This PR makes it possible for `rpc_v2::Storage::query_iter_paginated` to be "backpressured" which is achieved by having a channel where the result is sent back and when this channel is "full" we pause the iteration. The chainHead_follow has an internal channel which doesn't represent the actual connection and that is set to a very small number (16). Recall that the JSON-RPC server has a dedicate buffer for each connection by default of 64. - Because `archive_storage` also depends on `rpc_v2::Storage::query_iter_paginated` I had to tweak the method to support limits as well. The reason is that archive_storage won't get backpressured properly because it's not an subscription. (it would much easier if it would be a subscription in rpc v2 spec because nothing against querying huge amount storage keys) - `query_iter_paginated` doesn't necessarily return the storage "in order" such as - `query_iter_paginated(vec![("key1", hash), ("key2", value)], ...)` could return them in arbitrary order because it's wrapped in FuturesUnordered but I could change that if we want to process it inorder (it's slower) - there is technically no limit on the number of storage queries in each `chainHead_v1_storage call` rather than the rpc max message limit which 10MB and only allowed to max 16 calls `chainHead_v1_x` concurrently (this should be fine) - Iterate over 10 accounts on westend-dev -> ~2-3x faster - Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster - Fetch 1024 descendant values -> ~500x faster The reason for this is because as Josep explained in the issue is that one is only allowed query five storage items per call and clients has make lots of calls to drive it forward.. --------- Co-authored-by: command-bot <> Co-authored-by: James Wilson <james@jsdw.me>

niklasad1 · 2024-10-18T08:39:18Z

Yeah, ok I had another look and it was possible. Opened #6114 as well puuh :)

carlosala · 2024-10-18T09:27:28Z

yeah, cool!

rpc v2: rely backpressure Storage::query_iter

7f32fdf

niklasad1 commented Sep 17, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs Outdated Show resolved Hide resolved

Update substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs

c7d9a50

niklasad1 commented Sep 17, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs Outdated Show resolved Hide resolved

Update substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs

540daa6

niklasad1 commented Sep 17, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/tests.rs Outdated Show resolved Hide resolved

niklasad1 added 2 commits September 17, 2024 17:37

Update substrate/client/rpc-spec-v2/src/chain_head/tests.rs

423bc30

Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…

ef62794

…iter

niklasad1 added 3 commits September 18, 2024 11:29

cleanup

7351206

revert archive static limits

b963c8b

cargo fmt

1cb17f3

niklasad1 changed the title ~~WIP: rpc v2: rely backpressure Storage::query_iter~~ WIP: rpc v2: rely backpressure for chainHead_v1_storage Sep 18, 2024

niklasad1 added 4 commits September 19, 2024 09:59

add trait bound RawIter: Send

0c94a6b

Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…

1612813

…iter

cargo fmt

583ea58

remove unused import

486b246

niklasad1 changed the title ~~WIP: rpc v2: rely backpressure for chainHead_v1_storage~~ rpc v2: rely backpressure for chainHead_v1_storage Sep 19, 2024

niklasad1 commented Sep 19, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs Outdated Show resolved Hide resolved

Update substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs

67a37ce

niklasad1 commented Sep 19, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs Outdated Show resolved Hide resolved

Update substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs

05386e8

niklasad1 marked this pull request as ready for review September 19, 2024 09:09

niklasad1 added the T3-RPC_API This PR/Issue is related to RPC APIs. label Sep 19, 2024

niklasad1 commented Sep 19, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs Outdated Show resolved Hide resolved

niklasad1 commented Sep 19, 2024

View reviewed changes

niklasad1 requested a review from lexnv September 19, 2024 12:49

lexnv reviewed Sep 24, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/chain_head.rs Outdated Show resolved Hide resolved

lexnv reviewed Sep 24, 2024

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs Outdated Show resolved Hide resolved

lexnv approved these changes Sep 24, 2024

View reviewed changes

Update prdoc/pr_5741.prdoc

25051cd

Co-authored-by: James Wilson <james@jsdw.me>

jsdw approved these changes Oct 1, 2024

View reviewed changes

Merge branch 'master' into na-fix-rpc-storage-iter

4a8c5c3

niklasad1 requested a review from a team October 1, 2024 18:06

niklasad1 added 4 commits October 2, 2024 17:14

remove needless trait bounds

9152f9e

Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…

ec25f00

…iter

remove more needless trait bounds

bb251b0

remove more needless trait bounds again

49e013c

niklasad1 enabled auto-merge October 2, 2024 15:56

Merge branch 'master' into na-fix-rpc-storage-iter

4a70b0e

niklasad1 added this pull request to the merge queue Oct 2, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 2, 2024

niklasad1 added this pull request to the merge queue Oct 3, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 3, 2024

niklasad1 added this pull request to the merge queue Oct 3, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 3, 2024

niklasad1 added this pull request to the merge queue Oct 3, 2024

Merged via the queue into master with commit 3313163 Oct 3, 2024
217 checks passed

niklasad1 deleted the na-fix-rpc-storage-iter branch October 3, 2024 18:59

niklasad1 mentioned this pull request Oct 17, 2024

[stable 2409] Backport #5741 #6110

Open

niklasad1 mentioned this pull request Oct 18, 2024

[stable 2407] Backport #5741 #6114

Open

carlosala mentioned this pull request Oct 18, 2024

rpc v2: backpressure chainhead_v1_follow #6058

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc v2: backpressure chainHead_v1_storage #5741

rpc v2: backpressure chainHead_v1_storage #5741

niklasad1 commented Sep 17, 2024 •

edited

Loading

paritytech-cicd-pr commented Sep 17, 2024

niklasad1 Sep 19, 2024 •

edited

Loading

lexnv Sep 24, 2024

lexnv left a comment •

edited

Loading

jsdw left a comment

carlosala commented Oct 17, 2024 •

edited

Loading

niklasad1 commented Oct 17, 2024 •

edited

Loading

carlosala commented Oct 18, 2024

niklasad1 commented Oct 18, 2024

carlosala commented Oct 18, 2024

rpc v2: backpressure chainHead_v1_storage #5741

rpc v2: backpressure chainHead_v1_storage #5741

Conversation

niklasad1 commented Sep 17, 2024 • edited Loading

Benchmarks using subxt on localhost

paritytech-cicd-pr commented Sep 17, 2024

niklasad1 Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

lexnv Sep 24, 2024

Choose a reason for hiding this comment

lexnv left a comment • edited Loading

Choose a reason for hiding this comment

jsdw left a comment

Choose a reason for hiding this comment

carlosala commented Oct 17, 2024 • edited Loading

niklasad1 commented Oct 17, 2024 • edited Loading

carlosala commented Oct 18, 2024

niklasad1 commented Oct 18, 2024

carlosala commented Oct 18, 2024

niklasad1 commented Sep 17, 2024 •

edited

Loading

niklasad1 Sep 19, 2024 •

edited

Loading

lexnv left a comment •

edited

Loading

carlosala commented Oct 17, 2024 •

edited

Loading

niklasad1 commented Oct 17, 2024 •

edited

Loading