Deleted topics are recreated by Kafdrop after Kafka’s automatic preferred replica leader election

[eliska-n]

Under certain conditions, Kafdrop seems to recreate recently deleted topics after Kafka completes an automatic preferred replica leader election.

Setup: I run both Kafka and Kafdrop in Docker using the following docker-compose.yaml configuration:

services:
  kafdrop-1:
    environment:
      KAFKA_BROKERCONNECT: kafka-1:9092
      SERVER_PORT: 9000
      SERVER_SERVLET_CONTEXTPATH: /kafdrop
    image: obsidiandynamics/kafdrop:4.0.2
  kafka-1:
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://lmio-eliska-1:9092
      KAFKA_BROKER_ID: '1'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181/kafka
    image: confluentinc/cp-kafka:7.5.1
    volumes:
    - /data/ssd/kafka-1/data:/var/lib/kafka/data

In this configuration, Kafka is set to automatically create a topic once a client references it.

Steps to Reproduce:

• Use Kafdrop’s UI to delete a topic (e.g., received.system.generic).
• Observe Kafka’s logs, which indicate that the topic deletion process starts successfully.
• If you refresh the Kafdrop UI immediately after deletion, the topic remains deleted as expected.
• If Kafdrop remains inactive in the UI, Kafka eventually performs an automatic preferred replica leader election (in my case, every 5 minutes).
• After the election, refreshing Kafdrop’s UI causes the deleted topic to reappear in Kafka without its original messages or configurations.

When multiple topics are deleted at once and Kafdrop remains inactive, refreshing the UI after the leader election consistently recreates only the last deleted topic.

The deleted topics should remain deleted, even if the Kafdrop UI is refreshed after a leader election.

Here’s a snippet of the Kafka logs to illustrate the issue. Note the timeline of events from the topic deletion initiation to its unexpected recreation:
I used Kafdrop UI to delete topic received.system.generic and the deletion process started in Kafka.

[2024-11-07 09:50:01,763] INFO [Controller id=1] Starting topic deletion for topics received.system.generic (kafka.controller.KafkaController)

The deletion process ends by deleting last partition. Next, there is the election. At ~09:56 I simply refreshed Kafdrop in the browser and the topic was created.

[2024-11-07 09:51:01,864] INFO Deleted log for partition received.system.generic-0 in /var/lib/kafka/data/received.system.generic-0.6dde56bf0349486f9ed50f906a34d232-delete. (kafka.log.LogManager)
[2024-11-07 09:54:08,212] INFO [Controller id=1] Processing automatic preferred replica leader election (kafka.controller.KafkaController)
[2024-11-07 09:54:08,212] TRACE [Controller id=1] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
[2024-11-07 09:54:08,212] DEBUG [Controller id=1] Topics not in preferred replica for broker 1 HashMap() (kafka.controller.KafkaController)
[2024-11-07 09:54:08,212] TRACE [Controller id=1] Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController)
[2024-11-07 09:56:21,882] INFO Creating topic received.system.generic with configuration {} and initial partition assignment HashMap(0 -> ArrayBuffer(1)) (kafka.zk.AdminZkClient)
[2024-11-07 09:56:21,918] INFO [Controller id=1] New topics: [HashSet(received.system.generic)], deleted topics: [HashSet()], new partition replica assignment [Set(TopicIdReplicaAssignment(received.system.generic,Some(fSFH7SmkTji2rvpjA6gdbA),Map(received.system.generic-0 -> ReplicaAssignment(replicas=1, addingReplicas=, removingReplicas=))))] (kafka.controller.KafkaController)
[2024-11-07 09:56:21,918] INFO [Controller id=1] New partition creation callback for received.system.generic-0 (kafka.controller.KafkaController)

Note: If there’s no activity in Kafdrop between the deletion and the election, the last deleted topic is always recreated on a UI refresh. This issue persists even if there’s a significant gap in time (e.g., hours).

Note2: This report is based solely on observed behavior; I have not delved into Kafdrop’s internal logic. The automatic preferred replica leader election may or may not be directly related to the issue. However, in my testing, I noticed that the topic recreation consistently occurs after the election logs appear. This timing could be coincidental, but it appears regularly in this sequence.