Build the spak application in Intelij with java 11 and gradle. The examples uses the following Open Source Projects.
- Apache Spark version 3.1.1
- Apache Kafka
- Apache ZooKeeper
- MongoDB
- Windows 11
spark example uses a number of open source projects to work properly:
- [spark] - Open source big data framework
- [intellij Community edition] - Awesome java IDE editor
- [JDK] - Open source JDK.
Simple Hello world program to load TXT, CSV and JSON files
Merge two data frames and store into Mongo DB database
Produce the messages to Kafka topic using spring boot Kafka clients and consume in spark structured streaming API
Refer Spark, Kafka and Mongo installation Guides