VStream API: allow keyspace-level heartbeats to be streamed #16593

rohit-nayak-ps · 2024-08-13T21:21:14Z

Description

This PR adds an option to the VTGate VStream API to allow heartbeats created by vttablet's heartbeat writer to be streamed as regular FieldEvents/RowEvents for the sidecar _vt.heartbeat table.

flags := &vtgatepb.VStreamFlags{
	StreamKeyspaceHeartbeats: true,
}

Note that the vttablet heartbeats have to be enabled either through the throttler (which is on by default) or explicitly turning on the heartbeats using, for example, --heartbeat_enable, --heartbeat_interval 1s and --heartbeat_on_demand_duration 0

Related Issue(s)

Fixes: Feature Request: Binlog Timestamp Watermarking for VStream #16477

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

…l tables Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

codecov · 2024-08-14T08:39:06Z

Codecov Report

Attention: Patch coverage is 90.12346% with 8 lines in your changes missing coverage. Please review.

Project coverage is 68.98%. Comparing base (cc68dd5) to head (8fb0240).
Report is 34 commits behind head on main.

Files	Patch %	Lines
go/vt/vttablet/tabletserver/vstreamer/vstreamer.go	94.33%	3 Missing ⚠️
go/vt/vtgate/vstream_manager.go	75.00%	2 Missing ⚠️
...et/tabletmanager/vreplication/replica_connector.go	0.00%	1 Missing ⚠️
go/vt/vttablet/tabletserver/binlog_watcher.go	0.00%	1 Missing ⚠️
go/vt/vttablet/tabletserver/tabletserver.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16593      +/-   ##
==========================================
+ Coverage   68.85%   68.98%   +0.12%     
==========================================
  Files        1557     1562       +5     
  Lines      199891   200776     +885     
==========================================
+ Hits       137644   138505     +861     
- Misses      62247    62271      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…nes and the ones specified separately Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

proto/binlogdata.proto

proto/vtgate.proto

go/vt/vtgate/vstream_manager.go

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go

go/test/endtoend/vreplication/vstream_test.go

proto/binlogdata.proto

go/vt/vtgate/vstream_manager.go

mattlord · 2024-08-19T18:11:29Z

go/vt/vttablet/tabletserver/tabletserver.go

@@ -1171,7 +1171,7 @@ func (tsv *TabletServer) VStream(ctx context.Context, request *binlogdatapb.VStr
 	if err := tsv.sm.VerifyTarget(ctx, request.Target); err != nil {
 		return err
 	}
-	return tsv.vstreamer.Stream(ctx, request.Position, request.TableLastPKs, request.Filter, throttlerapp.VStreamerName, send)
+	return tsv.vstreamer.Stream(ctx, request.Position, request.TableLastPKs, request.Filter, throttlerapp.VStreamerName, send, request.Options)


Is it worth adding vstream options to support a variadic set of options (0-N) like we recently did for lock options and (workflow) server options? Then we don't need to pass this as nil in so many places: https://github.com/vitessio/vitess/blob/main/go/vt/vtctl/workflow/server_options.go

I prefer using strong typing unless functionally contraindicated. Especially for a grpc API.

While updating all usages can be tedious, it also helps review the parts of code where that API is being called: #16593 (comment)

mattlord

LGTM! I only had some minor comments. Let me know what you think and I'll come back to this quickly.

go/vt/vttablet/tabletserver/vstreamer/helper_event_test.go

mattlord · 2024-08-19T18:37:30Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

-func TestInternalTables(t *testing.T) {
+func TestOnlineDDLTables(t *testing.T) {


This is a clue that it might be better to explicitly call these sidecar[db]_tables rather than internal_tables. Although they're all internal tables so it's also fine as-is IMO.

yeah, I was also vacillating between internal and sidecar(db). internal is not precise because we could also term online ddl shadow tables and other artifacts as internal. I will let @deepthi also weigh in before changing it everywhere.

This seems to have been resolved.

@deepthi, the question was a broader one. Currently we use options/proto attributes like InternalTables, is_internal. Should we change that to SidecarDBTables and is_sidecardb_table?

We could either continue calling it Internal (keeping it generic, allowing for future specification of other tables than SidecarDB tables, like, I don't know, shadow tables in OnlineDDL for example) or lock it down to SidecarDB tables using that in the object naming instead.

internal is fine for now. Renaming proto fields is not difficult if we wanted to change it in the future.

mattlord · 2024-08-19T18:38:28Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

@@ -398,6 +398,79 @@ func TestMissingTables(t *testing.T) {
 	runCases(t, filter, testcases, startPos, nil)
 }

+// TestInternalTables tests streaming of sidecar db tables.


This comment for the function name is another clue that it might be better/clearer to call them SidecarDBTables rather than InternalTables.

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

deepthi · 2024-08-23T21:24:26Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

-func TestInternalTables(t *testing.T) {
+func TestOnlineDDLTables(t *testing.T) {


This seems to have been resolved.

go/vt/vttablet/tabletmanager/vreplication/external_connector.go

deepthi · 2024-08-23T21:35:28Z

go/vt/vttablet/tabletmanager/vreplication/replica_connector.go

@@ -73,5 +73,5 @@ func (c *ReplicaConnector) Close() error {
 }

 func (c *ReplicaConnector) VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error {
-	return c.vstreamer.Stream(ctx, startPos, nil, filter, throttlerapp.ReplicaConnectorName, send)
+	return c.vstreamer.Stream(ctx, startPos, nil, filter, throttlerapp.ReplicaConnectorName, send, nil)


Here again, based on how this is being used (only in PITR), we don't pass in the options.
Is it possible to replace usages of this connector with the tabletConnector?

Is it possible to replace usages of this connector with the tabletConnector?

I think so. Should be a separate PR though.

We should even think about deleting all the binlog server code in the existing PITR implementation. No one is using it.

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

mattlord

Nice work, @rohit-nayak-ps ! ❤️

github-actions bot added this to the v21.0.0 milestone Aug 13, 2024

rohit-nayak-ps added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication Component: Throttler Component: VTGate labels Aug 13, 2024

rohit-nayak-ps added 11 commits August 14, 2024 10:17

Add VStream flag to enable heartbeats

78687fc

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Add e2e test

657ccca

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Always send heartbeat table events on

1085d56

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Add IsInternal to FIELD/ROW events for sidecar tables

2af77cc

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Pass options through to VStreamer

0a24b85

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Fix unit tests for new Stream signature

6d55bb7

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Minor refactor

f00e37c

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Allow streaming of any internal tables. Add unit test. Update e2e test

25c92ef

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Don't error out while dropping tables in defer to account for interna…

fa40b47

…l tables Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Only expose keyspace heartbeats in VTGate API

d516f8b

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Fix vstream call for new signature in vdiff framework

731799a

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

rohit-nayak-ps force-pushed the rohit/vstream-vt-heartbeat branch from 68863d4 to 731799a Compare August 14, 2024 08:17

rohit-nayak-ps added 2 commits August 14, 2024 11:47

If streaming from a sidecar stream all tables, not just the special o…

22fade6

…nes and the ones specified separately Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Self-review

45edbb1

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

rohit-nayak-ps mentioned this pull request Aug 14, 2024

Feature Request: Binlog Timestamp Watermarking for VStream #16477

Closed

rohit-nayak-ps added 2 commits August 16, 2024 08:15

Self-review: e2e test case

ad21a8e

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Self-review

5221811

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

rohit-nayak-ps marked this pull request as ready for review August 16, 2024 08:41

rohit-nayak-ps requested review from deepthi, harshit-gangal, mattlord, derekperkins, systay and shlomi-noach as code owners August 16, 2024 08:41

rohit-nayak-ps requested review from timvaillancourt, GuptaManan100 and ajm188 as code owners August 16, 2024 08:41

rohit-nayak-ps mentioned this pull request Aug 18, 2024

VReplication: make flags workflow-specific and dynamically changeable #16583

Merged

5 tasks

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

proto/binlogdata.proto Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

proto/vtgate.proto Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

go/vt/vtgate/vstream_manager.go Outdated Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go Outdated Show resolved Hide resolved

mattlord reviewed Aug 19, 2024

View reviewed changes

Address review comments

06e7b17

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

deepthi reviewed Aug 23, 2024

View reviewed changes

rohit-nayak-ps added 2 commits August 24, 2024 19:26

Address review comments

2f63e6a

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

Address review comments: change IsInternal to IsInternalTable

8fb0240

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

mattlord approved these changes Aug 26, 2024

View reviewed changes

deepthi approved these changes Aug 27, 2024

View reviewed changes

deepthi merged commit bc6fb1c into vitessio:main Aug 27, 2024
132 checks passed

deepthi deleted the rohit/vstream-vt-heartbeat branch August 27, 2024 16:09

makinje16 mentioned this pull request Jan 16, 2025

VStream API [Backport]: Allow Keyspace-Level Heartbeats to be Streamed (#16593) slackhq/vitess#586

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VStream API: allow keyspace-level heartbeats to be streamed #16593

VStream API: allow keyspace-level heartbeats to be streamed #16593

rohit-nayak-ps commented Aug 13, 2024 •

edited by mattlord

Loading

codecov bot commented Aug 14, 2024 •

edited

Loading

mattlord Aug 19, 2024

rohit-nayak-ps Aug 26, 2024

mattlord left a comment

mattlord Aug 19, 2024

rohit-nayak-ps Aug 20, 2024

deepthi Aug 23, 2024

rohit-nayak-ps Aug 24, 2024

deepthi Aug 26, 2024

mattlord Aug 19, 2024

deepthi Aug 23, 2024

deepthi Aug 23, 2024

rohit-nayak-ps Aug 24, 2024 •

edited by deepthi

Loading

deepthi Aug 27, 2024

mattlord left a comment

		func TestInternalTables(t *testing.T) {
		func TestOnlineDDLTables(t *testing.T) {

VStream API: allow keyspace-level heartbeats to be streamed #16593

VStream API: allow keyspace-level heartbeats to be streamed #16593

Conversation

rohit-nayak-ps commented Aug 13, 2024 • edited by mattlord Loading

Description

Related Issue(s)

Checklist

codecov bot commented Aug 14, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohit-nayak-ps Aug 24, 2024 • edited by deepthi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord left a comment

Choose a reason for hiding this comment

rohit-nayak-ps commented Aug 13, 2024 •

edited by mattlord

Loading

codecov bot commented Aug 14, 2024 •

edited

Loading

rohit-nayak-ps Aug 24, 2024 •

edited by deepthi

Loading