Skip to content

Home

Jump to bottom
Shubhendu Madhukar edited this page Mar 21, 2021 · 21 revisions

Welcome to the kafka-service wiki!

WHAT

Kafka Service is a spring boot application built to simplify testing Kafka based applications. This application provides you with a configurable test producer, ability to post templated payloads (inspired by Handlebar.java and Wiremock's templating helpers). See HOW section for more details.

WHY

Most testing tools (Licensed such as Loadrunner and Cavisson Netstorm, or Open Sourced such as JMeter and Locust), are built with a primary focus on testing HTTP Based applications. Though all of the application mentioned can be used to send payloads to Kafka, they often don't provide great APIs for usability. Kafka Service was built to bridge that gap.

Using Kafka Service, you can now build your scripts the same way you'd build them if you were testing a HTTP web/microservices based applications. Your scripts will call Kafka Service APIs, which would in turn process your requests and post the messages to Kafka for you.

Example:

curl -X POST "http://localhost:8080/kafka/publish?topic=Test" -H "accept: */*" -H "Content-Type: application/json" -d "{\"message\":\"Hello World\"}"

This posts a message "Hello World" to a Kafka topic named "Test" to a Kafka cluster hosted locally on your machine. See HOW section for more details about configuring the application to post messages to a remote Kafka cluster, a SSL secured Kafka cluster, with templated messages and more.

HOW

Running the application locally

  • Begin by clone this repository
    • git clone https://github.com/fauxauldrich/kafka-service.git
  • There are several ways to run Kafka Service application on your local machine. One way is to execute the main method in the com.fauxauldrich.kafkaservice.KafkaServiceApplication class from your IDE.
  • Alternatively you can use the Spring Boot Maven plugin like so:
mvn spring-boot:run
  • Navigate to http://localhost:8080/swagger-ui.

Deploying the application

Kafka Service application can be deployed in several ways

  • Build the application first.
    • mvn clean install package
  • Simply run as a background service on a linux server
    • java -jar target/kafka-service-${VERSION}.jar
    • On windows, use utilities such as NSSM to create a service.
  • Or deploy as a docker container using the sample Dockerfile provided in the repository.
  • The generated docker image can also be pushed to a local docker registry and deployed on cloud platforms or on prem Kubernetes clusters.
  • To deploy on a Tomcat server, package the application as a war file.
    • Update the line <packaging>jar</packaging> in pom.xml as <packaging>war</packaging>
    • Build the application again: mvn clean package
    • Deploy the war file to a tomcat server by:
      • Placing the war file in tomcat/webapp directory; OR
      • Using Tomcat Manager GUI

Configuring the application

Following configurable properties can be updated as per your requirements (src/main/resources/application.properties):

  • spring.kafka.bootstrap-servers=127.0.0.1:9092
  • kafkaservice.trustore.location=~/truststore.jks
  • kafkaservice.trustore.password=Password@123

Endpoints

  • /kafka/publish - Posts a simple message to a Kafka topic with following details.
    • Query Parameters: topic, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
   "message": "string"
}
  • /kafka/publish/with-headers - Posts a message along with additional headers
    • Query Parameters: topic, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
  "message": "string",
  "headers": {
    "additionalProp1": "string",
    "additionalProp2": "string",
    "additionalProp3": "string"
  }
}
  • /kafka/publish/secure - Posts a simple message to a Kafka topic with following details. Use if connecting to Kafka cluster requires SSL authentication. Needs properties kafkaservice.trustore.location and kafkaservice.trustore.password to be updated with relevant details before deployment.
    • Query Parameters: topic, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
   "message": "string"
}
  • /kafka/publish/secure/with-headers: Same as /kafka/publish/secure but accepts additional headers
    • Query Parameters: topic, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
  "message": "string",
  "headers": {
    "additionalProp1": "string",
    "additionalProp2": "string",
    "additionalProp3": "string"
  }
}

Kafka Service uses Spring Kafka templates to send messages to any cluster. However the templates are not created at runtime on the invocation of APIs, instead they are created during application initialization. This helps with the performance, but compromises with modularity. Imagine a scenario where you want to post a message to a different cluster or update the truststore details, only solution would be to update the application.properties and redeploy the application.

If you don't want to redeploy the application every time you need to test a new cluster, following "Dynamic" APIs allow you to send a message to any cluster and any topic without updating the configs. They do so by creating a new producer every time the APIs are invoked and destroying the producer once message is sent. Please note that this might not be very scalable. But if you are testing with small TPS, go nuts.

  • /kafka/dynamic/publish
    • Query Parameters: topic, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
   "brokers": "string",
   "message": "string"
}
  • /kafka/dynamic/publish/with-headers
    • Query Parameters: topic, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
   "brokers": "string",
   "message": "string",
   "headers": {
     "additionalProp1": "string",
     "additionalProp2": "string",
     "additionalProp3": "string"
   }
}
  • /kafka/dynamic/publish/secure
    • Query Parameters: topic, TRUSTSTORE_LOCATION, TRUSTSTORE_PASSWORD, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
   "brokers": "string",
   "message": "string"
}
  • /kafka/dynamic/publish/secure/with-headers
    • Query Parameters: topic, TRUSTSTORE_LOCATION, TRUSTSTORE_PASSWORD, KEY(Optional), PARTITION_ID(Optional)
    • Body: 
      {
   "brokers": "string",
   "message": "string",
   "headers": {
     "additionalProp1": "string",
     "additionalProp2": "string",
     "additionalProp3": "string"
   }
}

Response Templating

You don't need to depend on your testing tool to parameterise your payload. Kafka Service comes with multiple inbuilt handlebars to help you with string manipulation. Following example explains it better:

Suppose you want to send following message to Kafka:

{
   "OrderCreatedAt": 1616320755763,
   "CustomerPhoneNuber": 1234567890
}

But you want to update these values for every message you send. You'd usually use your testing tool to generate random values and current timestamp for you. But you can also format POST body for any of the above APIs as:

{
  "message": "{ \"OrderCreatedAt\": {{now format='epoch'}}, \"CustomerPhoneNuber\": {{randomValue length=10 type='NUMERIC'}} }"
}

and you don't need to create parameters in your test tool, Kafka Service will handle that for you.

Refer to Handlebar documentation at:

  • StringHelpers (Handlebars 4.2.0 API)
  • Some of the handlebars are inspired by Wiremock, such as:
    • randomValue - eg: {{randomValue length=33 type='ALPHANUMERIC'}}
    • date - eg: {{date (parseDate '03/21/2021') offset='-1 days'}}
    • now (overwrites 'now' provided by StringHelpers in the documentation above) - eg: {{now offset='2 years' format='epoch'}}
    • parseDate - eg: {{date (parseDate '03/21/2021') offset='-1 days'}}
    • trim - eg: {{trim ' Text with whitespaces '}}
    • base64 - eg: {{base64 'text to encode'}} Or {{base64 'text to decode' decode=true}}
    • urlEncode - eg: {{urlEncode 'text to encode'}} Or {{urlEncode 'text to decode' decode=true}}
    • size - eg: {{size 'abcde'}}
    • pickRandom - eg: {{{pickRandom '1' '2' '3'}}}
Clone this wiki locally