Skip to content

Files

Latest commit

 

History

History

java

Java Apache Spark and Structured Streaming Examples

Examples Spark program run the Spark, Dataframe and Streaming

Build the spak application in Intelij with java 11 and gradle. The examples uses the following Open Source Projects.

  • Apache Spark version 3.1.1
  • Apache Kafka
  • Apache ZooKeeper
  • MongoDB
  • Windows 11

Tech

spark example uses a number of open source projects to work properly:

  • [spark] - Open source big data framework
  • [intellij Community edition] - Awesome java IDE editor
  • [JDK] - Open source JDK.

Examples

spark-hello

Simple Hello world program to load TXT, CSV and JSON files

mergedata-dframe

Merge two data frames and store into Mongo DB database

kafka-spark-structured-streaming

Produce the messages to Kafka topic using spring boot Kafka clients and consume in spark structured streaming API

Installation

Refer Spark, Kafka and Mongo installation Guides