Airflow Terraform

Goal

Easily provision resources on AWS for a highly scalable and secure airflow environment.

Overview

Terraform is used to provision resources in a VPC.
Airflow's services are deployed on an EC2 instance.
The Airflow metadata database is deployed on RDS (Postgres).
The Airflow message broker is deployed on Elastic Cache (Redis).
AWS Secrets Manager is used to securely feed environment variables such as airflow's fernet key into the EC2 instance.

Requirements

AWS Account
Terraform

Setup

Create an EC2 key pair or use an existing one aws ec2 create-key-pair --key-name MyKeyPair

Terraform Commands

First enter the Terraform directory cd terraform

See a plan of resources terraform plan
Provision resources terraform apply
Destroy resources terraform destroy