Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: tablet throttler deprecations in v21 #1801

Merged
merged 1 commit into from
Aug 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion content/en/docs/21.0/reference/features/tablet-throttler.md
Original file line number Diff line number Diff line change
Expand Up @@ -783,7 +783,6 @@ The primary throttler uses `CheckThrottler` gRPC calls on the replicas. Apps int
The throttler does also provide a HTTP endpoint for external apps such as `gh-ost` and `pt-online-schema-change`:

- `/throttler/check?app=<app-name>` is the equivalent of `vtctldclient CheckThrottler --app-name=<app-name>`.
- `/throttler/check-self`, is the equivalent of `vtctldclient CheckThrottler --scope="self"`.

### Metrics

Expand Down Expand Up @@ -876,6 +875,30 @@ Counter. Number of times throttler was probed via `CheckRequest` gRPC.

Counter. Number of times the throttler has requested a heartbeat lease. Correlated with `HeartbeatWrites` metric, and specifically when `--heartbeat_on_demand_duration` is set, this helps diagnose throttler/heartbeat negotiation and behavior.

## Deprecations

The following are deprecated in `v21`, and will be removed in `v22` or later:

### Flags

- The flags `--check-as-check-self` and `--check-as-check-shard` in `vtctldclient UpdateThrottlerConfig`. These flags were useful in the single-metric throttler. The new multi-metric design allows a per app configuration of not only a list of metrics, but also the scope of each metrics. For example, both these commands can be applied:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we show a deprecation message when people try to use these flags?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


```shell
$ vtctldclient UpdateThrottlerConfig --app-name "all" --app-metrics "lag,self/loadavg" commerce
$ vtctldclient UpdateThrottlerConfig --app-name "online-ddl" --app-metrics "lag,threads_running,shard/loadavg" commerce
```
Each applies different metrics to different apps, and each can assign a different scope to any specific metric.

### HTTP endpoints

- `/throttler/check-self` - use `vtctldclient CheckThrottler --scope-self` instead
- `/throttler/status` - use `vtctldclient GetThrottlerStatus` instead
- `/throttler/throttle-app` - use `vtctldclient UpdateThrottlerConfig --throttle-app=<name> ...` instead
- `/throttler/unthrottle-app` - use `vtctldclient UpdateThrottlerConfig --unthrottle-app=<name>` instead
- `/throttler/throttled-apps` - use `vtctldclient GetThrottlerStatus` instead

The endpoint `/throttler/check` is kept but unsupported. It is used by `gh-ost`. External use of this endpoint is discouraged.

## Notes

The throttler is originally derived from GitHub's [freno](https://github.com/github/freno). Over time, its design has significantly diverged from `freno`.
6 changes: 2 additions & 4 deletions content/en/docs/21.0/reference/vreplication/throttling.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,11 @@ To help address the above issues, VReplication uses the [tablet throttler](../..

### Target Throttling

On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the internal equivalent of `/throttler/check` when writing data to the shard's primary. This checks the MySQL replication lag on relevant replicas in the shard. The throttler will delay the VReplication writes of both table-copy and changelog events until the shard's replication lag is under the defined threshold (1s by default).
On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the tablet throttler when writing data to the shard's primary. This checks the MySQL replication lag on relevant replicas in the shard, and/or any other configured metrics. The throttler will delay the VReplication writes of both table-copy and changelog events until the shard's replication lag or other metrics are below their respective thresholds..

### Source Throttling

On the source side, VReplication only affects the single MySQL server it reads from, and has no impact on the overall shard. VStreamer, the source endpoint of VReplication, consults the equivalent of `/throttler/check-self`, which looks for replication lag on the source host.

As long as `check-self` fails — meaning that the replication lag is not within the defined threshold (1s by default) — VStreamer will not read table data, nor will it pull events from the changelog.
On the source side, VReplication only affects the single MySQL server it reads from, and has no impact on the overall shard. VStreamer, the source endpoint of VReplication, consults the tablet's throttler. By default this looks for replication lag on tablet's MySQL server. It may look for additional/other metrics per configuration. As long as any of the metrics exceeds its threshold, VStreamer will not read table data, nor will it pull events from the changelog.

### Impact of Throttling

Expand Down