How to connect to local (bare-metal, outside a container) kafka instance

[GloWorm0]

I'm trying to connect Kafka-ui to a Kafka cluster that is running locally on my machine, NOT via a Docker container. It will connect fine to other clusters that I am monitoring, but not to my local cluster. No matter what I try it reports the local cluster as offline.

In my Kafka config/server.properties I have the following entries:
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

I'm currently using the following docker-compose file:
version: '3.7'
services:
kafka-ui:
image: provectuslabs/kafka-ui
ports:
- "9080:8080"
environment:
KAFKA_CLUSTERS_0_NAME: Local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: host.docker.internal:9092
KAFKA_CLUSTERS_0_ZOOKEEPER: host.docker.internal:2181
KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: Local
KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: http://host.docker.internal:8093

I see the following in the log output:
kafka-ui_1 | 17:02:27.776 [kafka-admin-client-thread | adminclient-60] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster Local info

kafka-ui_1 | org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1636563747774, tries=1, nextAllowedTryMs=1636563747875) timed out at 1636563747775 after 1 attempt(s)

kafka-ui_1 | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

This is what I get in the ui:

As you can see, I can connect to other clusters on my network, just not my local cluster.

I've tried replacing host.docker.internal with my the local IP address of my machine and the IP address assigned by my VPN, and receive the same problem.

Is there some other configuration that I'm missing?

Thanks

[github-actions[bot]]

Hello there GloWorm0! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

[Haarolean]

Hey, so, you’re running kafka-ui in a container, but kafka itself is being run on bare metal. Is that right?

…

On 10 Nov 2021, at 20:13, github-actions[bot] ***@***.***> wrote:  Hello there GloWorm0! 👋 Thank you and congratulations 🎉 for opening your very first issue in this project! 💖 In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[GloWorm0]

Yes, that is correct.

[Haarolean]

@GloWorm0
Which platform are you running the app on?
How did you try passing the host.docker.internal?

[GloWorm0]

I'm running on macOS (10.15.7).

I'm not sure I understand. I just tried to use it when setting the environment variables for the container. So instead of KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: localhost:9092 (which obviously won't work because it's not running in the container) I tried KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: host.docker.internal:9092 (as well as actually using the IP.

[Haarolean]

It's not just that.
First of all, your docker-compose should have this:

services:
  kafka-ui:
    extra_hosts:
      - host.docker.internal:host-gateway
[...]

Secondly, your KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS should look like this: host.docker.internal:9092.
Moreover, take a look at your KAFKA_ADVERTISED_LISTENERS, it should be like this: PLAINTEXT://host.docker.internal:9092.

[GloWorm0]

I had tried the extra hosts at one point and it didn't help, but I will try again, perhaps I added it incorrectly. I'm relatively new to using Docker.

My KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS is currently host.docker.internal:9092. I will try again with extra_hosts added back.

Is KAFKA_ADVERTISED_LISTENERS a variable for kafka-ui? I don't see that listed in the list of environment variables. Kafka is not running in a docker container, so setting advertised.listeners in my server.properties to host.docker.internal would cause an error.

I'll try 'extra-hosts' again and see if I added it incorrectly the first time I tried.

[Haarolean]

I just tried that setup and it worked for me.

KAFKA_ADVERTISED_LISTENERS is kafka's variable.
Yeah, that's probably right. I tried that setup with kafka running in a container but I've accessed it externally via port forwarding, so, something close to your setup.

[GloWorm0]

Adding extra-hosts didn't change anything. Very frustrating. I can access other clusters, just not my local Kafka. I'm trying to evaluate this tool and would rather do some of the tests just on my local cluster, but I guess I won't be able to do that.

I'm sure it's something with my setup, but have no idea what it could be. Kafka is up and running and accessible, just not from within Docker.

I've run other simpler tests with containers not involving Kafka and can use hosts.docker.internal to access the local machine, just not with this container.

I'll have to see if I can get another container to interact with my local Kafka.

[Haarolean]

I'd also try to run a 'fair' instance of kafka on bare metal to try that thing out. I suppose the trick might be about KAFKA_ADVERTISED_LISTENERS.

[GloWorm0]

OK, I figured it out.

In my Kafka server.properties I had advertised.listeners=PLAINTEXT://localhost:9092. This only allow access via localhost. Once I changed it to my machine's IP address everything worked fine.

Thanks for the help.

[pejeio]

Hi @GloWorm0 , Thanks, I also changed it to my machine's IP address and everything worked fine.
However, I find it cumbersome to constantly update the config whenever my machine's IP address changes. Is there an alternative solution that would be more convenient for everyone involved?

[Haarolean]

@pejeio DNS could be used here, or docker additional hosts if you run the ui as a container.

[rdhmuhammad]

just want to add, i also set the listeners tolisteners=PLAINTEXT://0.0.0.0:9092along side with advertised.listeners=PLAINTEXT://{{my_ip_address}}:9092 to make it work.

[Haarolean]

You're welcome :) I'll convert this to a discussion to make it easily discoverable for others.

[mebigfatguy]

Hmm having the same problem, tried updating

advertised.listeners=PLAINTEXT://localhost:9092

with my IP, and still nothing. Can you please explain explicitly what needs doing for a noob?

[Haarolean]

@mebigfatguy hey, please share your configs :)

[mebigfatguy]

i have the default kafka configuration, directly from unzipping the download other than enabling the

advertised.listeners=PLAINTEXT://192.168.120.45:9092

line. i am running the default docker image of kafka-ui

Both kafka and kafka-ui are running locally

i can see the kafka-ui, but nothing is populated.

[Haarolean]

@mebigfatguy what's the config for kafka-ui?

[mebigfatguy]

i just ran this

docker run -p 8888:8080
-e KAFKA_CLUSTERS_0_NAME=local
-e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
-d provectuslabs/kafka-ui:latest

as was shown in the help. (had to change the public port because of confict)

[Haarolean]

@mebigfatguy the thing is, kafka:9092 is a hostname of another container, which you don't have. You have to pass host.docker.internal:port there, like described above.

[njavalgi03]

Hey,Everyone

I am trying to connect to my local kafka cluster but it won't connect. I tried all the above steps that was discussed in this thread but it still won't connect. Can anyone please help?

This is how my compose.yaml file looks:

version: '3.7'
services:
kafka-ui:
extra_hosts:

• "host.docker.internal:host-gateway"
  image: provectuslabs/kafka-ui
  ports:
• "9080:8080"
  environment:
  KAFKA_CLUSTERS_0_NAME: Local
  KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: host.docker.internal:2181
  KAFKA_CLUSTERS_0_ZOOKEEPER: host.docker.internal:2181
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://host.docker.internal:2181

Am I missing anything in this? Also, in service.properties file, I've added my machine IP in advertised.listeners.

[Haarolean]

@njavalgi03 which host OS are you on?