feat(shard-distributor): add SpectatorPeerChooser for shard-aware routing #7478
+365
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakobht
requested review from
3vilhamster,
Shaddoll,
davidporter-id-au,
demirkayaender,
dkrotx,
neil-xie,
sankari165,
shijiesheng and
taylanisikdemir
as code owners
jakobht
force-pushed
the
add-spectator-peer-chooser
branch
4 times, most recently
from
642ad52 to
1597904
Compare
jakobht
force-pushed
the
add-spectator-peer-chooser
branch
from
1597904 to
f4bb653
Compare
Implement a YARPC peer chooser that routes requests to the correct
executor based on shard ownership. This is the shard distributor
equivalent of Cadence's RingpopPeerChooser.
Flow:
1. Client calls RPC with yarpc.WithShardKey("shard-key")
2. Chooser queries Spectator for shard owner
3. Extracts grpc_address from owner metadata
4. Creates/reuses peer for that address
5. Returns peer to YARPC for connection
The peer chooser maintains a cache of peers and handles concurrent
access safely. It uses the x-shard-distributor-namespace header to
determine which namespace's spectator to query.
Dependencies:
- Requires spectator GetShardOwner to return metadata (see previous commit)
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
The tests have dependency issues with mock generation that need to be resolved separately. The peer chooser implementation is complete and functional. Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Tests cover: - Success path with peer creation - Peer reuse on subsequent calls - Error cases (missing shard key, namespace header, spectator not found) - Lifecycle methods (Start, Stop, IsRunning) - SetSpectators method Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
jakobht
force-pushed
the
add-spectator-peer-chooser
branch
from
f4bb653 to
85017c4
Compare
dkrotx
reviewed
Nov 28, 2025
| logger log.Logger | ||
| namespace string | ||
|
|
||
| mu sync.RWMutex |
Member
There was a problem hiding this comment.
nit: maybe comment that mu is exactly to cover changes in peers (and not, maybe spectators).
| // | ||
| // The ShardKey in the request is the actual shard key (e.g., workflow ID, shard ID), | ||
| // NOT the ip:port address. This is the key distinction from directPeerChooser. | ||
| func (c *SpectatorPeerChooser) Choose(ctx context.Context, req *transport.Request) (peer.Peer, func(error), error) { |
Member
There was a problem hiding this comment.
maybe it is already explained in the inrterface, but since comments are here, I think returning func(error) deserves an explanation. Especially since it's always either nil or [no-op] func(error) {}.
| } | ||
|
|
||
| // Extract GRPC address from owner metadata | ||
| grpcAddress, ok := owner.Metadata[grpcAddressMetadataKey] |
Member
There was a problem hiding this comment.
nit: a bit unexpected to see such specifics as grpcAddressMetadataKey here.
Comment on lines
+132
to
+145
| // Check if we already have a peer for this address | ||
| c.mu.RLock() | ||
| p, ok := c.peers[grpcAddress] | ||
| if ok { | ||
| c.mu.RUnlock() | ||
| return p, func(error) {}, nil | ||
| } | ||
| c.mu.RUnlock() | ||
|
|
||
| // Create new peer for this address | ||
| p, err = c.addPeer(grpcAddress) | ||
| if err != nil { | ||
| return nil, nil, yarpcerrors.InternalErrorf("failed to add peer for address %s: %v", grpcAddress, err) | ||
| } |
Member
There was a problem hiding this comment.
I suggest moving this logic to getOrCreatePeer sort-a thing.
Then it will be a single place dealing with c.mu + the usage [optimised get covered by RLock] should become more obvious.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR depends on #7476 (spectator return metadata) being merged first.
What changed?
Added
SpectatorPeerChooserthat implements YARPC's peer.Chooser interface to route requests to the correct executor based on shard ownership.Why?
Enable executor-to-executor communication in the shard distributor canary system. The peer chooser queries the Spectator to find which executor owns a shard, then routes requests to that executor's gRPC address.
How did you test it?
Unit tests
Potential risks
None
Release notes
Documentation Changes