Replies: 1 comment
-
|
This reads promising: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Working with Airflow version 3.1+ (3.1.7 currently).
I've come across this task parametrization feature in Airflow to reuse decorated tasks: https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#task-parameterization , e.g. to instantiate tasks multiple times with different task_ids. Even though this looks to be for portability between DAGs, or instantiation of multiple similar tasks within a single DAG, I am interested in the simple ability to override chosen attributes in the
@taskdecorator.Is it possible to couple this somehow with Params, the ones provided at trigger-time of a DAG, such that some attributes of the
@taskdecorator may be modified depending on what the user inputs at trigger-time?Motivation
We have a custom Executor which we specify in the
@taskdecorator and provide anexecutor_configattribute for configuration. This executor_config, among other things, includes which machine to run on and e.g. how many resources to allocate (particularly how many nodes to use in an HPC setting). The custom Executor then allocates these resources for the job to run. Many workflows are of the nature of being able to run with multiple such different configurations (especially varying the number of nodes), however, we are having trouble understanding if this .override functionality has the ability to work in tandem with the Params specified at trigger-time.I see something similar at: #46880, where a user wants to launch tasks on a user-specified data center at trigger-time, however their Airflow version is 2.x. Is this situation still the same, i.e. is the
task_instance_mutation_hookcluster policy the "right" way forward here, or is there a better way to achieve this?Could the solution be to implement custom operators with these specific fields (machine, number of nodes) as templatable, and then possibly create a custom decorator to align with the TaskFlow API syntax (or is that somehow easier said than done)?
Here seems to be another discussion where a user is expressing a similar desire to use Params to override some specific attributes of the task: #21627. In essence I guess my discussion is somewhat reviving those older ones in the context of a more recent Airflow version.
Beta Was this translation helpful? Give feedback.
All reactions