Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

chainHead: Limit ongoing operations #14699

Merged
merged 28 commits into from
Aug 15, 2023
Merged

chainHead: Limit ongoing operations #14699

merged 28 commits into from
Aug 15, 2023

Conversation

lexnv
Copy link
Contributor

@lexnv lexnv commented Aug 2, 2023 •
edited by niklasad1
Loading

This PR adds support for limiting the number of ongoing operations:

  • at most 16 operations can be executed by a chainHead_follow subscription (items in chainHead_storage count each as a operation)
  • methods return limitReached when capacity cannot be reserved (or subscription ID is invalid or stale)
  • chainHead_storage discards elements when there's not enough capacity
  • struct ChainHeadConfig is added to have granular control over the limits of the API
  • reserving capacity and permits is a slim wrapper over Arc<Mutex<usize>> with a configurable maximum and a RAII style release of capacity on drop
  • the capacity permit is propagated to the BlockGuard
  • unit-test for low-level OngoingOperations and PermitOperations
  • integration-test for limit propagation to methods via ensure_operation_limits_works()

This builds on top of: #14692
Closes: #14640

// @paritytech/subxt-team

lexnv added 19 commits July 31, 2023 18:54
@lexnv
chainHead/api: Make storage/body/call pure RPC methods
4787dba 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Add mpsc channel between RPC methods
8050811 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/subscriptions: Extract mpsc::Sender via BlockGuard
3b30eb3 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/subscriptions: Generate and provide the method operation ID
d5305a9 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Generate chainHead_body response
50c840d 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Generate chainHead_call response
6c5940c 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Generate chainHead_storage responses
589efa8 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Propagate responses of methods to chainHead_follow
8062837 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/tests: Adjust chainHead_body responses
4f7b445 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/tests: Adjust chainHead_call responses
53fcd99 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/tests: Adjust chainHead_call responses
167dd72 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/tests: Ensure unique operation IDs across methods
35aca6d 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/events: Remove old method events
008fa22 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/subscriptions: Add limit helper
6c30aef 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/subscription: Expose limits to BlockGuard
a29d677 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/tests: Adjust testing to ongoing operations
a33d80c 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Make limits configurable via ChainHeadConfig
40a73fb 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/tests: Adjust testing to ChainHeadConfig
eb15293 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/tests: Ensure operation limits discards items
a026fe0 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv lexnv added A0-please_review Pull request needs code review. C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit B1-note_worthy Changes should be noted in the release notes T2-API This PR/Issue is related to APIs. labels Aug 2, 2023
@lexnv lexnv self-assigned this Aug 2, 2023
Base automatically changed from lexnv/chainhead_extend_resp to master August 8, 2023 18:13
@lexnv
Merge remote-tracking branch 'origin/master' into lexnv/chainhead_limits
4e4cca8 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Improve documentation
defc196 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Rename OngoingOperations -> LimitOperations
4cacbc1 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead: Rename reserve -> reserve_at_most
09c743c 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
Copy link
Contributor Author

lexnv commented Aug 10, 2023

bot rebase

@paritytech-processbot
Copy link

Rebased

@lexnv lexnv requested a review from skunert August 10, 2023 13:00
@lexnv lexnv mentioned this pull request Aug 11, 2023
Merged
@lexnv
chainHead: Use duration const instead of u64
ab481e8 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
chainHead/subscription: Use tokio::sync::Semaphore for limits
09e7372 
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
@lexnv
Merge remote-tracking branch 'origin/lexnv/chainhead_limits' into lex…
8954c33 
…nv/chainhead_limits
skunert
skunert reviewed Aug 15, 2023
View reviewed changes
Copy link
Contributor

@skunert skunert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Only some small comments.

client/rpc-spec-v2/src/chain_head/chain_head.rs Show resolved Hide resolved
client/rpc-spec-v2/src/chain_head/subscription/inner.rs Outdated Show resolved Hide resolved
@lexnv @skunert
Update client/rpc-spec-v2/src/chain_head/subscription/inner.rs
a3df109 
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
@lexnv lexnv merged commit a79a2af into master Aug 15, 2023
3 checks passed
@lexnv lexnv deleted the lexnv/chainhead_limits branch August 15, 2023 12:17
Ank4n pushed a commit that referenced this pull request Aug 20, 2023
@lexnv @skunert @Ank4n
chainHead: Limit ongoing operations (#14699)
981de06 
* chainHead/api: Make storage/body/call pure RPC methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Add mpsc channel between RPC methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Extract mpsc::Sender via BlockGuard

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Generate and provide the method operation ID

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_body` response

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_call` response

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_storage` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Propagate responses of methods to chainHead_follow

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_body` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_call` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_call` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Ensure unique operation IDs across methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/events: Remove old method events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Add limit helper

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscription: Expose limits to `BlockGuard`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust testing to ongoing operations

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Make limits configurable via `ChainHeadConfig`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust testing to `ChainHeadConfig`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Ensure operation limits discards items

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Improve documentation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Rename `OngoingOperations` -> `LimitOperations`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Rename reserve -> reserve_at_most

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Use duration const instead of u64

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscription: Use tokio::sync::Semaphore for limits

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update client/rpc-spec-v2/src/chain_head/subscription/inner.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: parity-processbot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
@crystalin crystalin mentioned this pull request Oct 20, 2023
Closed
@JesseAbram JesseAbram mentioned this pull request Oct 26, 2023
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@niklasad1 niklasad1 niklasad1 approved these changes

@skunert skunert skunert approved these changes

Assignees

@lexnv lexnv

Labels
A0-please_review Pull request needs code review. B1-note_worthy Changes should be noted in the release notes C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit T2-API This PR/Issue is related to APIs.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[RPC-Spec-V2] Limit ongoing operations
3 participants
@lexnv @skunert @niklasad1