Data Engineering Labs

A hands-on collection of data engineering projects. The module structure is inspired by the Data Engineering Zoomcamp from DataTalks.Club. All implementations are original work.

Refer to the modules below for covered topics and tools.

Modules

Module 1: Data ingestion

• Python ingestion with polars and pandas
• Rust data ingestion
• ELT Ingestion with Airbyte
• data load tool (dlt)
• IaC with Terraform (Google Cloud Platform)

Module 2: Workflow orchestration

• Workflow orchestration with Airflow 3.x
• Workflow orchestration with Airflow 2.x
• Workflow orchestration with Prefect

Module 3: Lakehouses & Data Warehouse

• BigQuery Data Warehouse
• StarRocks Query Engine

Module 4: Analytics engineering

• BigQuery and dbt
• Snowflake and dbt
• Databricks and dbt
• Redshift and dbt
• ClickHouse and dbt
• PostgreSQL and dbt
• DuckDB and dbt
• Data visualization with Metabase
• Data visualization with Superset

Module 5: Batch processing

• PySpark 4.x + Spark Connect
• PySpark 3.x + Spark Connnect
• Spark + Kotlin API
• Spark (Scala)

Module 6: Stream processing

• PyFlink
• Stream processing with Kafka, ksqlDB and Kotlin
• Kafka Streams with ksqlDB

Extras

• LakeHouse with Delta, Iceberg, Hive
• Data Catalog with DataHub