Skip to content

Commit

Permalink
v0.0.4 (#3)
Browse files Browse the repository at this point in the history 
Added optional HTTP BASIC authentication
Added defensive code around configuration
Refactored producer and consumer code
General code cleanup
  • Loading branch information
@dhoard
dhoard authored Feb 15, 2023
1 parent b4451c9 commit 509a409
Show file tree
Hide file tree
Showing 15 changed files with 694 additions and 382 deletions.
19 changes: 19 additions & 0 deletions .github/workflows/build.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Build
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up JDK 11
uses: actions/setup-java@v3
with:
java-version: '11'
distribution: 'temurin'
cache: maven
- name: Build
run: mvn clean package
30 changes: 18 additions & 12 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,31 @@

Kafka synthetic test application to measure round trip time.

**Because the message only contains the produce timestamp (Unix epoch), latency will be greater than a real Kafka use case**

## Build

```
cd <project directory>
mvn clean package
```

## Run
## Kafka Topic Configuration

You need to create a unique test topic per application instance with enough partitions to span all brokers

**Notes**

- This application uses manual partition assignment


For **each** instance of the application, create topic with a partition count that is equal to the number of Kafka brokers.
- Example topic name is `kafka-synthetic-test-<id>`... where `<id>` matches the `id` in `test.properties`

- Example name is `kafka-synthetic-test-<id>`... where `<id>` matches the `id` in `test.properties`

- Example retention time is `600,000` ms (10 minutes) (Old messages are skipped)
- Example retention time is `300,000` ms (5 minutes) (Old messages are skipped)


## Run


Copy `test.properties` and edit to match your environment
Expand All @@ -37,23 +48,18 @@ Example output:
```
# HELP kafka_synthetic_test_round_trip_time Kafka synthetic test round trip time
# TYPE kafka_synthetic_test_round_trip_time gauge
kafka_synthetic_test_round_trip_time{id="us-west-1.32",bootstrap_servers="cp-3:9092",topic="kafka-synthetic-test",partition="2",} 24.0
kafka_synthetic_test_round_trip_time{id="us-west-1.32",bootstrap_servers="cp-3:9092",topic="kafka-synthetic-test",partition="1",} 10.0
kafka_synthetic_test_round_trip_time{id="us-west-1.32",bootstrap_servers="cp-3:9092",topic="kafka-synthetic-test",partition="0",} 16.0
kafka_synthetic_test_round_trip_time{id="source-10.0.0.1",bootstrap_servers="cp-3:9092",topic="kafka-synthetic-test",partition="0",} 6.0
kafka_synthetic_test_round_trip_time{id="source-10.0.0.1",bootstrap_servers="cp-3:9092",topic="kafka-synthetic-test",partition="1",} 7.0
kafka_synthetic_test_round_trip_time{id="source-10.0.0.1",bootstrap_servers="cp-3:9092",topic="kafka-synthetic-test",partition="2",} 9.0
```

**Notes**

- **This application uses manual partition assignment. You need to create a test topic per application instance**


- A test message is sent to every partition based on the configured `period.ms` value


- A negative value indicates that a metric hasn't been updated within the configured `metric.expiration.period.ms` value


- Because the message only contains the produce time, latency will be greater than a real Kafka use case

# FOR DEMO PURPOSES ONLY - NOT SUPPORTED

2 changes: 1 addition & 1 deletion kafka-synthetic-test.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ then
exit 1
fi

java -jar target/kafka-synthetic-test-0.0.3.jar "${1}"
java -jar target/kafka-synthetic-test-0.0.4.jar "${1}"
219 changes: 102 additions & 117 deletions pom.xml
View file
Original file line number Diff line number Diff line change
@@ -1,125 +1,110 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.github.dhoard</groupId>
<artifactId>kafka-synthetic-test</artifactId>
<version>0.0.3</version>
<groupId>com.github.dhoard</groupId>
<artifactId>kafka-synthetic-test</artifactId>
<version>0.0.4</version>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>org.slf4j:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
<filter>
<artifact>ch.qos.logback:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
<filter>
<artifact>org.apache.kafka:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
<filter>
<artifact>org.xerial.snappy:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
<filter>
<artifact>org.lz4:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
<filter>
<artifact>com.github.luben:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
<filter>
<artifact>io.prometheus:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.github.dhoard.kafka.KafkaSyntheticTest</Main-Class>
<Build-Number>1.0</Build-Number>
</manifestEntries>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
<includes>
<include>**/kafka-synthetic-test.properties</include>
</includes>
</resource>
<resource>
<directory>src/main/resources</directory>
<filtering>false</filtering>
<excludes>
<exclude>**/kafka-synthetic-test.properties</exclude>
</excludes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.github.dhoard.kafka.KafkaSyntheticTest</Main-Class>
<Build-Number>1.0</Build-Number>
</manifestEntries>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.30</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.3.2</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.16.0</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
<version>0.16.0</version>
</dependency>
</dependencies>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.30</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.3.2</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.16.0</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
<version>0.16.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>31.1-jre</version>
</dependency>
</dependencies>

</project>
Loading

0 comments on commit 509a409

Please sign in to comment.