Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hive: tablet id to node id resolver + a per-node caching service #9385

Open
snaury opened this issue Sep 17, 2024 · 0 comments
Open

Hive: tablet id to node id resolver + a per-node caching service #9385

snaury opened this issue Sep 17, 2024 · 0 comments
Assignees

Comments

@snaury
Copy link
Member

snaury commented Sep 17, 2024

We want to quickly determine where a particular tablet id is running (or will be running soon), and currently only hive is the authoritative source of that info. This may be used to quickly select where an external clients should redirect their requests for maximum locality. It's currently possible to do via pipe cache, but with multiple drawbacks:

  1. Pipe cache has to connect the pipe first, which is usually cheap when idle (for leader tablets), but it may have an extremely negative effect during request storms, when cache is cold and pipes are not connected yet.
  2. Pipe cache connects to some (closest) instance when used with followers, so it cannot be used to gather node ids for all followers. Worse, follower pipes are not cheap, since pipe cache uses short lived follower pipes and keeps reconnecting periodically (to rediscover better instances that may have appeared since the previous connection).
  3. It's already a service that has to receive and forward all tablet messages, adding more traffic that is not directly related to requests may affect latency.
  4. Connected pipes require connected interconnect, which may cause all nodes to connect to all other nodes unnecessarily (it's ok when it's done by a local kqp session, it has to gather and forward results anyway, but for external redirects it's entirely avoidable).

Thus we want some HiveCache service that takes a list of tablet ids and returns corresponding node ids (possibly with all followers), ideally without any network round trips from a coherent local cache. A rough sketch of what it does:

  1. Connects to a root hive and caches a coherent mapping from tablet range to an owner hive
  2. For each relevant hive (where relevant means we needed that info at least once) it connects and caches a mapping of some basic info for each tablet (leader and follower nodes, maybe something else very basic). This info must be easy to cache without unnecessary traffic on reconnects (we don't want to redownload millions of tablets on every hive restart/disconnect), so it should be versioned with rollup using easily computed deltas.
  3. For each relevant hive it must cache which tablet ids are actually in use (using some sparse bitmaps), so it is possible to detect deleted tablets very efficiently (including subscriptions and notifications as soon as a given tablet is deleted). See Tablet pipes with CheckAliveness may be too slow for volatile transactions #4797.

This would allow us to efficiently and locally answer a bunch of questions about tablets (with info that is almost always up-to-date). Ideally there should be a C++ object for getting cached info without expensive cross-actor messaging.

@snaury snaury self-assigned this Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant