Skip to content

Commit

Permalink
[doc] Document Asynchronous for lookup changelog-producer
Browse files Browse the repository at this point in the history
  • Loading branch information
JingsongLi committed Nov 29, 2023
1 parent 40d373b commit 2a7a01f
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions docs/content/maintenance/write-performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ Option `'changelog-producer' = 'lookup' or 'full-compaction'`, and option `'full
large impact on write performance, if it is a snapshot / full synchronization phase you can unset these options and
then enable them again in the incremental phase.

If you find that the input of the job shows a jagged pattern in the case of backpressure, it may be imbalanced work
nodes. You can consider turning on [Asynchronous Compaction]({{< ref "#asynchronous-compaction" >}}) to observe if the
throughput is increased.

## Parallelism

It is recommended that the parallelism of sink should be less than or equal to the number of buckets, preferably equal. You can control the parallelism of the sink with the `sink.parallelism` table property.
Expand Down Expand Up @@ -80,6 +84,9 @@ sort-spill-threshold = 10
This configuration will generate more files during peak write periods and gradually merge into optimal read
performance during low write periods.

In the case of `'changelog-producer' = 'lookup'`, by default, the lookup will be completed at checkpointing, which
will block the checkpoint. If you want an asynchronous lookup, you can enable `'changelog-producer.lookup-wait' = 'true'`.

### Number of Sorted Runs to Pause Writing

When the number of sorted runs is small, Paimon writers will perform compaction asynchronously in separated threads, so
Expand Down

0 comments on commit 2a7a01f

Please sign in to comment.