Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: reduce partition ingest time #2194

Open
ruojieranyishen opened this issue Feb 17, 2025 · 0 comments
Open

feat: reduce partition ingest time #2194

ruojieranyishen opened this issue Feb 17, 2025 · 0 comments
Labels
type/enhancement Indicates new feature requests

Comments

@ruojieranyishen
Copy link
Collaborator

ruojieranyishen commented Feb 17, 2025 •
edited
Loading

Feature Request

Is your feature request related to a problem? Please describe:
During the ingest phase, partitions block writes. Through testing, this time is approximately 30 seconds, while the rocksDB ingest time is around 200 milliseconds. The main reason is that during the ingest phase, the time interval for the meta to send RPC_BULK_LOAD is too long (once every 10 seconds).

Single - node ingest concurrency level Table - level ingest time (ms) Minimum partition - level ingest time (ms) Maximum partition - level ingest time (ms) Average partition - level ingest time (ms) Highest Rocksdb ingest time (ms) Average Rocksdb ingest time (ms)
8 470006 15007 30075 27244 618 459.06
16 240005 19978 30002 28652 580 482.43
100 40004 29967 29986 29978 620 509.65

Describe the feature you'd like:
reduce partition ingest time
@ruojieranyishen ruojieranyishen added the type/enhancement Indicates new feature requests label Feb 17, 2025
ruojieranyishen pushed a commit to ruojieranyishen/incubator-pegasus that referenced this issue Feb 17, 2025
feat: reduce partition ingest time (apache#2194)
414d2b0 
The main reason is that during the ingest phase, the time
between meta's RPC_BULK_LOAD transmissions is too long
(once every 10 seconds). Shortening this time interval
can cut down the partition write-blocking time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement Indicates new feature requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ruojieranyishen