Added docker image with 2-node Kafka cluster

[pulyaevskiy]

Hey,

Single node Kafka image is really nice. But it is not possible to run very good tests with single-node setup since there is a lot of leader discovery logic for topic-partitions and consumers.

So I created 2-node cluster image based on spotify/kafka and thought it would be good to contribute back.

This PR contains working Dockerfile in kafkacluster and instructions in README.md. However there is couple things left:

• There is no public build like spotify/kafkacluster, obviously.
• Options ADVERTISED_HOST and ADVERTISED_PORT might need to change. At least I would probably just get rid of ADVERTISED_PORT since it doesn't make sense for multi-node setup in single container (brokers use different ports).

But first, I was wondering if this is something maintainers are interested in?

[amarcionek]

Not a maintainer, but I would be interested in it.

[amarcionek]

I'm curious as to the details behind this comment: "But it is not possible to run very good tests with single-node setup since there is a lot of leader discovery logic for topic-partitions and consumers." We have issues with this original container where if we stop the container and start it, or the host system needs to be rebooted (thus the container is stopped/started) that the kafka/zookeeper doesn't respond anymore and we need to completely remove the container and re-run it. Its not horrible for a testing environment, although we'd like to be able to test the durability of the topics and its not possible to do so.

Our initial thought was these were zookeeper issues, not kafka ones. We found some issues in kafka and they seem centered around the way zookeeper works. See KAFKA-1387 and KAFKA-1451 Both bugs report as resolved, but I don't know about that.

In any case, our thought was that we just need to do cluster zookeeper, not kafka. In your patch, you have two brokers and one zookeeper.

One idea we had for modification of the spotify image was to allow for an n way configured cluster across n containers (instead of one.) So what we'd need is a way to specify the zookeeper clustering (if on the same host, each container would need unique backend ports (see RunningReplicatedZooKeeper) and same fo the broker ports, although I'm not sure what else kafka needs to interact with the zookeeper itself, other than client side settings.

Anyway, we're going down this route now and will post status if we get anywhere.

[pulyaevskiy]

@amarcionek

kafka/zookeeper doesn't respond anymore and we need to completely remove the container and re-run it

I'm using two-node container based on the one in this PR with Docker Toolbox and Travis-CI and haven't ran at any issues so far. Even after I stop/start the container.

[ralph-tice]

isn't this better solved via docker-compose ? and i think 3 node is a minimum ?