Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate orchestration to apache airflow / google cloud composer #14

Open
maxheld83 opened this issue Apr 12, 2021 · 0 comments
Open

migrate orchestration to apache airflow / google cloud composer #14

maxheld83 opened this issue Apr 12, 2021 · 0 comments
Assignees

Comments

@maxheld83
Copy link
Contributor

medium-term, this would be the right tool for the job.
it's also open source, so there's very little lock-in.

For now, to keep the scope of this manageable, we can try to accomplish the same with cron jobs on github actions.

However, medium-term, apache airflow is a better tool:

  • it does not pollute the actions log with non-code changes (it's technically possible to run gh actions for any reason, but non-commit events still feel awkward).
  • airflow's DAG natively "understand" the dependencies between the ETL steps; the actions runtime doesn't, we have to retrofit it by returning statuses from whichever services we query
  • airflow's DAGs support more complicated graphs; these may be possible in gh actions, but are hard to reason about / understand.
@maxheld83 maxheld83 moved this to Todo in max's scrum(ish) Nov 3, 2021
@maxheld83 maxheld83 self-assigned this Nov 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant