Client

What are some good resources for getting started with Kafka?

If you're new to Apache Kafka, the introduction and design sections of the Apache documentation are an excellent place to start.

The Confluent blog has a lot of good technical information - Jay Kreps's guide to building a streaming platform covers many of the core concepts again, but with a focus on Kafka's role at a company-wide scale. Also noteworthy are Ben Stopford's microservices blogposts for his unique take on the relationship between applications and data.

What is Confluent Platform?

Confluent Platform is a distribution of Apache Kafka. A good comparison of Apache Kafka, Confluent OSS and Confluent Enterprise can be found here.

Can I use the .NET Client to check if my Kafka cluster is down?

This question is not as straight forward as it sounds. What does 'cluster down' mean? Is that all brokers down? Or just the leaders for the partitions we care about? Does it include the replicas? If all brokers are down is this maybe just a configuration error on the client, or a temporary networking problem?

If we propagate broker state information via the client, should we then make partition leader information available, and maybe consumer coordinator information? Should we provide everything you need to re-implement the capability the client already provides?

We take the approach that as a user you shouldn't care. You configure the message.timeout.ms and message.max.retries settings and let the client take care of the rest. At the end of the day, it typically boils down to the question: what amount of time will I allow for a message to be sent before it is deemed outdated?

How many connections should we expect to see from the .NET client into the kafka brokers?

What factors determine the connection count? (#brokers, #topics, #partitions, #client consumer instances, other?)

Refer to: https://github.com/edenhill/librdkafka/wiki/FAQ#number-of-broker-tcp-connections

The number of open connections is determined by the number of brokers. The client writes / reads data directly from the broker that is the leader for the partition of interest and commonly a client will require connections to all brokers.

The worst case number of connections held open by librdkafka is: cnt(bootstrap.servers) + cnt(brokers in Metadata response). The minimum number of connections held open is cnt(brokers in Metadata response). Currently, librdkafka holds connections open to all brokers whether or not they are needed. In the future, we plan to enhance librdkafka so that disused connections are not maintained.

Where can I find a list of all the configuration properties?

Configuration parameters

You should generally avoid creating multiple Consumer or Producer instances in your process if possible:

• Each client will maintain a connection with all the brokers in the cluster.
• If you create multiple clients, data won't be batched as efficiently as if you had just one.
• All operation on client are thread safe (with some small exceptions documented in the API).

You can create multiple clients if you want. If you do, they won't share any data between them.

Should I create multiple consumer to distribute partitions?

You won't gain throughput by creating multiple clients in a single process to dispatch process - you should rather subscribe to all your topics in one consumer, and dispatch processing of message to worker tasks (TODO: examples and benchmark of those solutions).

You should consider deploying your application on multiple machines - this way, you are more reliable and bandwidth between your Kafka cluster and the worker machine won't be the bottleneck. This is the way to do with kafka to process data from multiple partitions.