A flexible data integration tool to help nonprofits connect to their data collection tools and ERP systems

Ready to go Apache Airflow stack for docker

Data Engineering using Apache Airflow

An EL pipeline built with Apache Airflow that downloads a file from the web uploads it to Google Cloud Storage, and creates an external table in BigQuery for data storage and analysis.

A comprehensive data engineering pipeline, orchestrates data workflows with Apache Airflow, Python, Kafka, Zookeeper, Spark, and Cassandra. Containerized using Docker: to deploy and scale effortlessly. This Etsy API Data Pipeline extracts, processes, and analyzes Etsy marketplace data—retrieving product listings, shop details, and reviews.

Invoice de-duplication via Azure Form Recognition, OpenAI, Apache Airflow and Redis Enterprise VSS

ETL pipeline using Apache Airflow

Setting up airflwo with best practices

Load data from the Million Song Dataset into a final dimensional model in RedShift utilizing Apache Airflow.

Custom XCom backend implementation for Airflow, with data serialization to S3

An extension enabling the monitoring of Apache Airflow DAGs directly from Jupyter notebooks. Tailored for developers and data scientists, it simplifies tracking specific DAGs, reduces unnecessary friction, and allows severity levels setup for failed DAGs.

The project will utilize Airflow to orchestrate and manage the data pipeline as it creates and terminates an EMR transient cluster to save on cost. Apache Spark will transform data, and the final dataset will be loaded into Snowflake.

A repository to learn Apache Airlfow by integrating it with other technologies.

Technology blogging website from Siby Abin. Talks about dataengineering, aws, spark, python, airflow and more

ETL Mini Project using Apache Airflow

This is the simple implementation of Apache airflow in a Kafka Project

Airflow sandbox