Apache Flink for Java Developers Workshop

Table of Contents

• 📚 Workshop Description & Learning Objectives
• 💻 Technical Prerequisites
• 🔧 Required Software and Setup
  • 1️⃣ Docker & Docker Compose (Mandatory)
  • 2️⃣ Confluent Cloud Account (Mandatory)
  • 3️⃣ Java 21 Installed
  • 4️⃣ Git Installed
  • 5️⃣ IDE with Gradle Support
  • 6️⃣ Gradle Installed via Wrapper (No Need for Local Installation)
  • 7️⃣ Quick Setup for macOS Users with Homebrew
• 🌐 Network & System Requirements
• ⚙️ Optional but Recommended
  • 1️⃣ Terraform Installed
  • 2️⃣ Basic Understanding of Terraform and IaC (Infrastructure as Code)
  • 3️⃣ Confluent CLI
  • 4️⃣ jq
• 📌 Pre-Workshop Setup Tasks

📚 Workshop Description & Learning Objectives

You were tasked with building a real-time platform, but traditional tools couldn’t handle the massive data streams, leading to lagging performance and frustrated users. Deadlines were looming, and the team needed a breakthrough.

And when you learned about Apache Flink.

We will explore how the DataStream API allowed efficient real-time data processing and how the Table API and SQL features simplified complex queries with familiar syntax. Testing became more straightforward, and managing the application state was no longer a headache.

You’ll learn:

• Harnessing the DataStream API: Process unbounded data streams efficiently to make your applications more responsive.

• Unlocking Table API and SQL: Use SQL queries within Flink to simplify data processing tasks without learning new languages.

• Effective Testing Strategies: Implement best practices for testing Flink applications to ensure your code is robust and reliable.

• Stateful Stream Processing: Manage application state effectively for complex event processing and real-time analytics.

By the end of this talk, you will be equipped to tackle real-time data challenges. Whether you’re building analytics dashboards, event-driven systems, or handling data streams, Apache Flink can be the game-changer you’ve been searching for.

💻 Technical Prerequisites

1. Basic Programming Knowledge – Familiarity with Java or Scala (ugh) (Flink supports both but not for long).

2. Understanding of Stream Processing Concepts – Awareness of real-time data pipelines, event-driven architectures, and streaming frameworks.

3. SQL Proficiency – Basic understanding of SQL for working with Flink’s Table API.

4. Linux/macOS Command Line Experience – Ability to execute terminal commands and navigate a Unix-like environment.

🔧 Required Software and Setup

1️⃣ Docker & Docker Compose (Mandatory)

• Why? Flink and Kafka components will be containerized.

• Download & Install: Docker Website

• macOS Alternative: OrbStack (recommended for better performance)

2️⃣ Confluent Cloud Account (Mandatory)

• Why? Required for Kafka-based streaming exercises.

• Sign Up & Get API Keys: Confluent Cloud

3️⃣ Java 21 Installed

• Why? Apache Flink will run on Java 21.

• Download Java 21 via SDKMAN: SDKMAN

• Verify Installation: Run java -version in the terminal.

4️⃣ Git Installed

• Why? For cloning repositories and working with project files.

• Download Git: Git Website

5️⃣ IDE with Gradle Support

• Why? Recommended for Flink development.

• Download IntelliJ IDEA (Recommended): IntelliJ IDEA

• Download VS Code (Alternative): VS Code

• Confluent Extension for VSCode: https://marketplace.visualstudio.com/items?itemName=confluentinc.confluent-vscode[link]

6️⃣ Gradle Installed via Wrapper (No Need for Local Installation)

• Why? The workshop will use the Gradle Wrapper, eliminating the need for manual installation.

• Gradle Docs: Gradle Wrapper Documentation

7️⃣ Quick Setup for macOS Users with Homebrew

• Why? Simplifies the installation of all required dependencies.

• How? This project includes a Brewfile for managing dependencies.

• Setup: Run make setup-mac to install all required dependencies using Homebrew.

• Update: Run make update-brew-deps to update dependencies.

🌐 Network & System Requirements

1. Stable Internet Connection – Required for downloading dependencies and connecting to Confluent Cloud.

2. 8GB+ RAM Recommended – Running Flink, and other services may require significant memory.

3. Sufficient Disk Space (At Least 10GB Free) – For Docker images, logs, and data processing.

⚙️ Optional but Recommended

1️⃣ Terraform Installed

• Why? Useful for automated infrastructure setup in Confluent Cloud.

• Download Terraform: Terraform Website

2️⃣ Basic Understanding of Terraform and IaC (Infrastructure as Code)

• Why? If Terraform scripts are used, a fundamental knowledge of how it works would be beneficial.

• Terraform Getting Started Guide: Terraform Tutorials

3️⃣ Confluent CLI

• Why? The workshop will use the commands in the Confluent CLI to get useful information about new Confluent infrastructure.

• Download and Install: Confluent CLI Installation Instructions

4️⃣ jq

• Why? The workshop will use jq to build configuration files used to demonstrate the Confluent Flink Table API.

• Download and Install: jq Download Instructions

📌 Pre-Workshop Setup Tasks

1. Sign up for Confluent Cloud & Configure API Keys – Ensure access credentials are available before the workshop.

2. Clone the Workshop Repository – The repo will include pre-built examples and configuration files (GitHub link will be shared before the workshop).

3. Set Up Environment Variables – Configure JAVA_HOME and authentication variables for Confluent Cloud.

4. Run a Simple Docker-Based Flink Job – Validate that the environment is correctly configured.