Skip to content

Commit

Permalink
Docs: tablet throttler deprecations in v21 (#1801)
Browse files Browse the repository at this point in the history
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
  • Loading branch information
shlomi-noach authored Aug 27, 2024
1 parent a7a0919 commit f1e1870
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 5 deletions.
25 changes: 24 additions & 1 deletion content/en/docs/21.0/reference/features/tablet-throttler.md
Original file line number Diff line number Diff line change
Expand Up @@ -783,7 +783,6 @@ The primary throttler uses `CheckThrottler` gRPC calls on the replicas. Apps int
The throttler does also provide a HTTP endpoint for external apps such as `gh-ost` and `pt-online-schema-change`:

- `/throttler/check?app=<app-name>` is the equivalent of `vtctldclient CheckThrottler --app-name=<app-name>`.
- `/throttler/check-self`, is the equivalent of `vtctldclient CheckThrottler --scope="self"`.

### Metrics

Expand Down Expand Up @@ -876,6 +875,30 @@ Counter. Number of times throttler was probed via `CheckRequest` gRPC.

Counter. Number of times the throttler has requested a heartbeat lease. Correlated with `HeartbeatWrites` metric, and specifically when `--heartbeat_on_demand_duration` is set, this helps diagnose throttler/heartbeat negotiation and behavior.

## Deprecations

The following are deprecated in `v21`, and will be removed in `v22` or later:

### Flags

- The flags `--check-as-check-self` and `--check-as-check-shard` in `vtctldclient UpdateThrottlerConfig`. These flags were useful in the single-metric throttler. The new multi-metric design allows a per app configuration of not only a list of metrics, but also the scope of each metrics. For example, both these commands can be applied:

```shell
$ vtctldclient UpdateThrottlerConfig --app-name "all" --app-metrics "lag,self/loadavg" commerce
$ vtctldclient UpdateThrottlerConfig --app-name "online-ddl" --app-metrics "lag,threads_running,shard/loadavg" commerce
```
Each applies different metrics to different apps, and each can assign a different scope to any specific metric.

### HTTP endpoints

- `/throttler/check-self` - use `vtctldclient CheckThrottler --scope-self` instead
- `/throttler/status` - use `vtctldclient GetThrottlerStatus` instead
- `/throttler/throttle-app` - use `vtctldclient UpdateThrottlerConfig --throttle-app=<name> ...` instead
- `/throttler/unthrottle-app` - use `vtctldclient UpdateThrottlerConfig --unthrottle-app=<name>` instead
- `/throttler/throttled-apps` - use `vtctldclient GetThrottlerStatus` instead

The endpoint `/throttler/check` is kept but unsupported. It is used by `gh-ost`. External use of this endpoint is discouraged.

## Notes

The throttler is originally derived from GitHub's [freno](https://github.com/github/freno). Over time, its design has significantly diverged from `freno`.
6 changes: 2 additions & 4 deletions content/en/docs/21.0/reference/vreplication/throttling.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,11 @@ To help address the above issues, VReplication uses the [tablet throttler](../..

### Target Throttling

On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the internal equivalent of `/throttler/check` when writing data to the shard's primary. This checks the MySQL replication lag on relevant replicas in the shard. The throttler will delay the VReplication writes of both table-copy and changelog events until the shard's replication lag is under the defined threshold (1s by default).
On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the tablet throttler when writing data to the shard's primary. This checks the MySQL replication lag on relevant replicas in the shard, and/or any other configured metrics. The throttler will delay the VReplication writes of both table-copy and changelog events until the shard's replication lag or other metrics are below their respective thresholds..

### Source Throttling

On the source side, VReplication only affects the single MySQL server it reads from, and has no impact on the overall shard. VStreamer, the source endpoint of VReplication, consults the equivalent of `/throttler/check-self`, which looks for replication lag on the source host.

As long as `check-self` fails — meaning that the replication lag is not within the defined threshold (1s by default) — VStreamer will not read table data, nor will it pull events from the changelog.
On the source side, VReplication only affects the single MySQL server it reads from, and has no impact on the overall shard. VStreamer, the source endpoint of VReplication, consults the tablet's throttler. By default this looks for replication lag on tablet's MySQL server. It may look for additional/other metrics per configuration. As long as any of the metrics exceeds its threshold, VStreamer will not read table data, nor will it pull events from the changelog.

### Impact of Throttling

Expand Down

0 comments on commit f1e1870

Please sign in to comment.