This set up is merely a demonstration for pushing logs to Logstash which then pushes the logs to a Kafka topic.
This set up is based on kafka-docker using the single-broker configuration.
- Docker needs to be installed.
- docker-compose needs to be installed.
Ensure that the following ports are not in use:
- 2181
- 9092
- 10000
Run:
docker-compose up -d
That's it! You are all set!
To see it in action:
-
Open up a terminal session and run:
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh --zookeeper zookeeper:2181 --topic logstash_logs
-
Open up a second terminal session and run:
curl -X POST -d @test.json http://localhost:10000 --header "Content-Type:application/json"
Ensure that
curl
is installed. -
You should be able to see a concise HTTP request in the form of an json object coming through in the first terminal session.
Now you can stream logs to localhost:10000 and see it in Kafka.
The configurations for Logstash is in the logstash_pipeline and logstash_settings folders.
You can modify the settings before running docker-compose up -d
.
In this demonstration, a Http input plugin and a Kafka output plugin using the json codec are used.