Configure multiple producers and consumer in properties (configuration for multiple clusters in spring.kafka properties)

[coen-jansen-ah]

If you are configuring producers or consumers for different clusters or with specific configurations, you have to add a custom property block and need programmatic configuration of these properties. This makes for a messie and complex application configuration file. From what I can see, it is easy to add this feature. That's why I was surprised it does not yet exist. I would like to see the configuration tree change something like this:

   `spring: 
	  kafka:
	    # Level 1: Global Properties (Applied to all consumers and producers)
	    enabled: true
	    bootstrap-servers: localhost:9092
	    security-protocol: SASL_SSL

	    # Global Structured Kafka Properties
	    properties:
	      specific.avro.reader: true
	      schema.registry.url: http://localhost:8081

	    # Global SASL Configuration
	    sasl:
	      mechanism: OAUTHBEARER
	      login-callback-handler-class: org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
	      jaas-config: >
	        org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required
	        clientId='${SPN_CLIENT_ID:placeholder}'
	        clientSecret='${SPN_CLIENT_SECRET:placeholder}'
	        scope='${KAFKA_BROKER_SCOPE:placeholder}';

	    # Global Listener Configuration
	    listener:
	      type: batch
	      concurrency: 3
	      ack-mode: BATCH
  
	    # Global Retry Configuration
	    retry:
	      max-attempts: 3
	      initial-interval: 1000ms
	      multiplier: 2.0
	      max-interval: 10000ms
	      retryable-exceptions: 
	        - java.lang.RuntimeException
	        - org.springframework.dao.RecoverableDataAccessException
    
	    # Global DLT Configuration
	    dlt:
	      suffix: .ERROR

	    # Level 2 & 3: Consumer-Specific Shared Configurations
	    consumers-configurations:
	      security-protocol: PLAINTEXT # Override global for consumers if needed
	      dlt-enabled: true
	      groupId: multi-kafka-default-group
	      kafka:
	        auto-offset-reset: earliest
	        max-poll-records: 500
	        enable-auto-commit: false
	        key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
	        value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer

	    # Level 2 & 3: Producer-Specific Shared Configurations
	    producers-configurations:
	      kafka:
	        acks: all
	        retries: 5
	        key-serializer: org.apache.kafka.common.serialization.StringSerializer
	        value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer

	    # Level 4: Specific Consumer Instances
	    consumers:
	      consumer-a:
	        beanName: consumerA # Custom bean name
	        topics: 
	          - consumer-a-topic
	        groupId: consumer-group-id # Overrides defaults
	        kafka:
	          client-id: consumer-a-client-id
	        retry:
	          max-attempts: 5 # Specific retry override
      
	      consumer-b:
	        topics: 
	          - consumer-b-topic
	        groupId: consumer-b-group-id
	        dlt-enabled: false # Disable DLT for this specific consumer
	        listener:
	          concurrency: 1 # Single threaded for ordering

	    # Level 4: Specific Producer Instances
	    producers:
	      producer-a:
	        beanName: ProducerA
	        kafka:
	          client-id: producer-a-client-id
	        dlt-enabled: true
	        dlt-properties:
	          kafka:
	            acks: 1 # Faster acks for DLT
      	    producer-b:
               beanName: ProducerB`

I have created a starter that implements this new property structure. You can find it in my Github: https://github.com/chjansen/MultiKafka

[artembilan]

I think there is some misunderstanding between Spring for Apache Kafka and Spring Boot.
All those spring.kafka properties are part of Spring Boot logic.
And that one is for microservices auto-configuration based on some conventional properties.
If you talk about "multi"-Kafka, then it is already not a Microservice, and you are free to do whatever you need with existing API from Spring for Apache Kafka.

Your project link gives us 404 to see some details of your idea.
Either way it feels like this should go to Spring Boot since neither of those properties are handled here in Spring for Apache Kafka.

[coen-jansen-ah]

So I need to present this idea to the Spring Boot team? Excuses for the not working link. The repo was private. It is public now. for example: we have a microservice that listens to two topics and then enriches the data and publishes it to a Kafka topic. So very much a microservice. As far as I am concerned.

[artembilan]

producers or consumers for different clusters or with specific configurations

I understand your use-case and fully agree that it is a real one.
But unfortunately, that is not how Spring Boot works: we can auto-configure exactly one cluster for all the consumers and/or producers.
The rest of complexity over configuration properties looks for me like an overhead for the framework.

Yes, it is better to share your idea with Spring Boot team since we don't have here in Spring Kafka any API on the matter and our team does not have those deep knowledge of Spring Boot to judge properly.

Thanks for understanding!

[coen-jansen-ah]

Thank you for your reaction.