If you're looking for Airflow videos from the 2022 edition, check the 2022 cohort folder.
Vicnenzo's DE Zoomcamp Prefect Repo - For Week 2
- What is a Data Lake
- ELT vs. ETL
- Alternatives to components (S3/HDFS, Redshift, Snowflake etc.)
- Video
- Slides
- What is orchestration?
- Workflow orchestrators vs. other types of orchestrators
- Core features of a workflow orchestration tool
- Different types of workflow orchestration tools that currently exist
- What is Prefect?
- Installing Prefect
- Prefect flow
- Creating an ETL
- Prefect task
- Blocks and collections
- Orion UI
- Flow 1: Putting data to Google Cloud Storage
- Flow 2: From GCS to BigQuery
- Parametrizing the script from your flow
- Parameter validation with Pydantic
- Creating a deployment locally
- Setting up Prefect Agent
- Running the flow
- Notifications
- Scheduling a deployment
- Flow code storage
- Running tasks in Docker
- Using Prefect Cloud instead of local Prefect
- Workspaces
- Running flows on GCP
TBA
Did you take notes? You can share them here.
- Add your notes here (above this line)
Most of these notes are about Airflow, but you will most likely find them useful too.
- Notes from Alvaro Navas
- Notes from Aaron Wright
- Notes from Abd
- Blog post by Isaac Kargar
- Blog, notes, walkthroughs by Sandy Behrens
- Add your notes here (above this line)