Clarify behavior of Snuba subscription consumers

[AbdulazizTamkeen]

Problem Statement

I’m trying to understand the behavior of Snuba subscription consumers in a self-hosted Sentry deployment.

While monitoring Kafka consumer groups, the subscription consumers appear to not advance the current offset, and in some cases the end offset is not shown, even though:

• The application is working correctly
• Events and subscriptions are functioning as expected

I have a basic understanding of Kafka and consumer groups, and the general Kafka documentation is clear, but it’s not obvious:

• What these subscription consumers are expected to consume
• Whether static offsets or missing end offsets are expected behavior
• How to correctly interpret their state when troubleshooting

This makes it difficult to determine whether their behavior is healthy or misconfigured.

This is example of a consumer in kafka-ui:

Solution Brainstorm

Add documentation that explains:

• What Snuba subscription consumers are responsible for
• Which topics they consume from and how
• Why offsets may appear static or missing
• What “healthy” behavior looks like for these consumers
• Any differences between subscription consumers and other Snuba consumers (for example, event consumers)

[stayallive]

I did notice this too, but I also saw the following.

• snuba-events-subscriptions-consumers maps to the snuba-commit-log topic
• snuba-eap-items-subscriptions-consumers maps to the snuba-items-commit-log topic

From what I can tell from inspecting the messages this looks like what you'd expect. Logs. Probably for (internal) metrics and I don't think they have any benefit for self-hosters that I can find at the moment.

Those two topics also contain an estimated of 20M messages on my instance according to the Kafka UI and are about 2MB on disk. So, not a very resource intensive deal luckily.

I found the following:

Alerts on Errors are generated by the Errors Subscription Consumer. This is synchronized consumer that consumes both the main events topic and the snuba-commit-log topic so it can proceed in lockstep with the main consumer.
The synchronized consumer then produces alerts by querying Clickhouse and produces the result on the result topic.

(here: https://github.com/getsentry/snuba/blob/0d480f1eae687c940ad6e6da436f3a0b7b94e713/docs/source/architecture/overview.rst#L121)

So this does sound like it might have something to do with alerts which I'm not a 100% sure all work (have not tested every type). Also not sure it's expected to advance the consumer groups or that it is expected to lag like this.

So I concur we either are missing something, something is broken or we just don't understand it 😅