Kafka Native

Kafka broker (and Zookeeper) compiled to native using Quarkus and GraalVM.

Project Structure

• quarkus-kafka-server-extension: Quarkus extension including for compiling Kafka Server to native using GraalVM.
• quarkus-zookeeper-server-extension: Quarkus extension including for compiling Zookeeper Server to native using GraalVM.
• kafka-server: Quarkus application starting a Kafka server using the kafka-server-extension. Compiles to JVM and native executable.
• zookeeper-server: Quarkus application starting a Kafka server using the zookeeper-server-extension. Compiles to JVM and native executable.
• kafka-native-test-container: Test containers starting a single-node Kafka broker using the native-compiled kafka-server and a single-node zookeeper using the native-compiled zookeeper-server. Includes integration tests.

Building the project

mvn install

Running kafka in dev mode

You can run kafka in dev mode that enables live coding using:

cd kafka-server
mvn compile quarkus:dev

Starts a single-node Kafka broker listening on PLAINTEXT://9092. Uses ./target/log-dir as log directory.

Running zookeeper in dev mode

You can run zookeeper in dev mode that enables live coding using:

cd zookeeper-server
mvn compile quarkus:dev

Starts a single-node zookeeper listening on 2181.

Packaging and running the application

The application can be packaged using the following on either the kafka-server (or zookeeper-server) directory:

mvn package

It produces the quarkus-run.jar file in the target/quarkus-app/ directory. Be aware that it’s not an über-jar as the dependencies are copied into the target/quarkus-app/lib/ directory.

The application is now runnable using java -jar target/quarkus-app/quarkus-run.jar.

Creating native executables

You can create a native executable using the following either the kafka-server (or zookeeper-server) directory:

mvn package -Pnative

Or, if you don't have GraalVM installed, you can run the native executable build in a container using:

mvn package -Pnative -Dquarkus.native.container-build=true

You can then execute your native executable with: ./target/kafka-server-1.0.0-SNAPSHOT-runner (or ./target/zookeeper-server-1.0.0-SNAPSHOT-runner)

Creating a container from native executable

You can create a container from the native executable using:

mvn package -Dnative -Dquarkus.native.container-build=true -Dquarkus.container-image.build=true

The container images will be built with tags quay.io/ogunalp/kafka-native:1.0.0-SNAPSHOT and quay.io/ogunalp/zookeeper-native:1.0.0-SNAPSHOT

If you want to reuse the existing native executable:

mvn package -Dnative -Dquarkus.native.reuse-existing=true -Dquarkus.container-image.build=true

In case your container runtime is not running locally, use the parameter -Dquarkus.native.remote-container-build=true instead of -Dquarkus.native.container-build=true.

Then you can run the docker image using:

docker run -p 19092:9092 -it --rm -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:19092 quay.io/ogunalp/kafka-native:1.0.0-SNAPSHOT

Configuring the Kafka broker

By default, the kafka-server application configures an embedded Kafka Kraft server for a single node cluster.

Following configuration options are available:

Key	Description	Default
server.kafka-port	External listener port	9092
server.internal-port	Internal listener port	9093
server.controller-port	Controller listener port	9094
server.delete-dirs-on-close	Whether to delete log-dir on application close	false
server.host	Hostname of listeners	`` (empty string)
server.cluster-id	Provide cluster-id, generated if empty
server.properties-file	Path to server.properties file
server.auto-configure	Automatically configure server properties, if false only server.properties is respected	true
kafka.log.dir	Path to log-dir directory, will create the directory if	./target/log-dir
kafka.advertised.listeners	Override advertised.listeners
kafka.zookeeper.connect	When configured the kafka broker starts in zookeeper mode	``
kafka.*	Override broker properties

You can set configuration options using Java system properties, e.g.

java -Dserver.delete-dirs-on-close=true \
  -Dserver.properties-file=server.properties \
  -Dkafka.advertised.listeners=SSL://localhost:9092 -jar ./target/quarkus-app/quarkus-run.jar

Or environment variables, e.g.

docker run -it --rm -p 19092:9092 \
  -v $(pwd):/conf \
  -e SERVER_PROPERTIES_FILE=/conf/server.properties \
  -e KAFKA_ADVERTISED_LISTENERS=SASL_PLAINTEXT://localhost:19092 \
  quay.io/ogunalp/kafka-native:1.0.0-SNAPSHOT