Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

README for doing local airflow dev using aws-mwaa-local-runner #475

Merged
merged 2 commits into from
Aug 7, 2023
Merged

README for doing local airflow dev using aws-mwaa-local-runner #475

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f92ee24
README for doing local airflow dev using aws-mwaa-local-runner
barbarahui Jul 5, 2023
b7cdb45
Update README.md
barbarahui Aug 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions airflow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Airflow Development

## Set up `aws-mwaa-local-runner`

AWS provides the [aws-mwaa-local-runner](https://github.com/aws/aws-mwaa-local-runner) repo, which provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally via use of a Docker container. We have forked this repository and made some small changes to enable us to use local-runner while keeping our dags stored in this repository. (See this slack thread for more info: https://apache-airflow.slack.com/archives/CCRR5EBA7/p1690405849653759)

To set up this dev environment, first clone the repo locally:

```
git clone git@github.com:ucldc/aws-mwaa-local-runner.git
```

Then, modify `aws-mwaa-local-runner/docker/.env`, setting the following env vars to wherever the directories live on your machine, for example:

```
DAGS_HOME="/Users/username/dev/rikolti/airflow/dags"
PLUGINS_HOME="/Users/username/dev/rikolti/airflow/plugins"
REQS_HOME="/Users/username/dev/rikolti/airflow"
STARTUP_HOME="/Users/username/dev/rikolti/airflow"
```

These env vars are used in the `aws-mwaa-local-runner/docker/docker-compose-local.yml` script (and other docker-compose scripts) to mount the relevant directories containing Airflow DAGs, requirements, and plugins files into the docker container.

Then, follow the instructions in the [README](https://github.com/ucldc/aws-mwaa-local-runner/#readme) to build the docker image, run the container, and do local development.


Loading