Skip to content

feat(microservices): add topic-consumers option to server-kafka#16555

Open
suuuuuuminnnnnn wants to merge 4 commits intonestjs:masterfrom
suuuuuuminnnnnn:feat/kafka-topic-consumers
Open

feat(microservices): add topic-consumers option to server-kafka#16555
suuuuuuminnnnnn wants to merge 4 commits intonestjs:masterfrom
suuuuuuminnnnnn:feat/kafka-topic-consumers

Conversation

@suuuuuuminnnnnn
Copy link
Contributor

PR Checklist

Please check if your PR fulfills the following requirements:

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Other... Please describe:

What is the current behavior?

ServerKafka uses a single KafkaJS consumer subscribed to all registered topics, so a slow handler on one topic can effectively block message processing on other topics (topic-level backpressure). Partition-level concurrency (partitionsConsumedConcurrently) does not address cross-topic blocking.

Issue Number: #14590

What is the new behavior?

Adds an opt-in topicConsumers: boolean option for Kafka microservices.

When topicConsumers: true:

  • Creates a separate consumer per registered topic and subscribes each consumer to exactly one topic
  • Uses {groupId}-{topic} as the consumer group id (independent offset tracking per topic)
  • Disconnects all per-topic consumers in parallel on close()
  • Logs a deduped warning when the composed group id appears invalid (whitespace or >255 chars)
  • Avoids mutating options.run when building per-topic run() options

Default remains false, preserving the existing single-consumer behavior.

Does this PR introduce a breaking change?

  • Yes
  • No

Note: enabling topicConsumers: true creates new consumer groups ({groupId}-{topic}), so migrating an existing deployment may require offset management.

Other information

Tests

npm test -- packages/microservices/test/server/server-kafka.spec.ts

All tests pass, including new unit tests for topicConsumers mode.

@coveralls
Copy link

Pull Request Test Coverage Report for Build 157b7e61-c1ed-4ba3-99e3-02b51ffa7962

Details

  • 46 of 50 (92.0%) changed or added relevant lines in 1 file are covered.
  • 59 unchanged lines in 5 files lost coverage.
  • Overall coverage decreased (-0.03%) to 89.815%

Changes Missing Coverage Covered Lines Changed/Added Lines %
packages/microservices/server/server-kafka.ts 46 50 92.0%
Files with Coverage Reduction New Missed Lines %
packages/microservices/server/server-kafka.ts 2 86.21%
packages/microservices/server/server-grpc.ts 4 85.39%
packages/microservices/client/client-redis.ts 16 79.84%
packages/microservices/server/server-mqtt.ts 16 83.04%
packages/microservices/server/server-redis.ts 21 76.15%
Totals Coverage Status
Change from base Build 3000a4b9-5b8f-4abf-b1d5-fd69b0426434: -0.03%
Covered Lines: 7487
Relevant Lines: 8336

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants