Application to terminate unused topics in Kafka
❗ NOTE: For now, dry-run is enabled by default, so to delete any unused topics, dry-run must be explicitly disabled.
The goal is to terminate (delete) unused topics in Kafka. Unused topics are topics that match the following three conditions:
- Not internal topic used internally by Kafka components
- Topic without consumers
- Inactive consumers are eventually cleaned up automatically by Kafka
- Topic that is empty
This project is heavily inspired by kafka-topic-cleaner but that project seems abandoned for a while, so we decided to create something new.
The application is a simple Spring Boot application with a scheduled cleanup-job that will run at a fixed (configurable) rate. The job will query the configured Kafka cluster for unused topics and delete (or just log; if dry-run enabled) the found unused topics.
As a Spring Boot application almost any aspect of the application runtime can be customized. The defaults should normally be sufficient, but if required please consult the Spring Boot common application properties for reference.
Configuration using environment variables is recommended. Please see Spring Boot externalized configuration to understand how to configure Spring Boot application properties using environment variables.
The application uses the Kafka admin client provided by
Spring for Apache Kafka to interact with the Kafka cluster.
Available properties are properties starting with spring.kafka in
Spring Boot integration properties.
As minimum, the spring.kafka.bootstrap-servers must be set.
Example set as environment variable:
SPRING_KAFKA_BOOTSTRAP_SERVERS=broker-1:9091,broker-2:9091,broker-3:9091
The application provides a few application-specific properties. See ApplicationProperties for an overview and application.yaml for the current default values.
The application provides a Prometheus endpoint with all application metrics enabled by default.
In addition to the standard JVM and Spring Boot metrics, the application provides some application specific metrics:
# HELP topic_deleted_total Number of topics deleted
# TYPE topic_deleted_total counter
topic_deleted_total N.0