-
Notifications
You must be signed in to change notification settings - Fork 213
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add elasticsearch concurrency tags for Airflow (#3921)
* Update sensor utils to get list of dag_ids matching a specified tag * Appropriately tag ES dags with their concurrency group, and use new sensor utils * Update dag docs * Automatically exclude the running DAG from the ids to check * Fix bad merge * Fix bad dag_id change, don't make db_restore wait unnecessarily * Only pass required context, use set * Rename variable for clarity * Split concurrency groups out into ES and DB groups * Clarify concurrency_tag variable name * Fix typo in dag docs
- Loading branch information
Showing
13 changed files
with
279 additions
and
184 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
from common.constants import PRODUCTION, STAGING | ||
|
||
|
||
# These DagTags are used to identify DAGs which should not be run concurrently | ||
# with one another. | ||
|
||
# Used to identify DAGs for each environment which affect the Elasticsearch cluster | ||
# and should not be run simultaneously | ||
PRODUCTION_ES_CONCURRENCY_TAG = "production_elasticsearch_concurrency" | ||
STAGING_ES_CONCURRENCY_TAG = "staging_elasticsearch_concurrency" | ||
|
||
# Used to identify DAGs which affect the staging API database in such a | ||
# way that they should not be run simultaneously | ||
STAGING_DB_CONCURRENCY_TAG = "staging_api_database_concurrency" | ||
|
||
ES_CONCURRENCY_TAGS = { | ||
PRODUCTION: PRODUCTION_ES_CONCURRENCY_TAG, | ||
STAGING: STAGING_ES_CONCURRENCY_TAG, | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.