Argo workflows design discussion

[kcpevey]

Context

The way to submit workflows to argo in nebari is to create a python script that you want to run on an argo pod, and then run some python code to submit that script. We have to send a conda run command to the docker container to run the script.

This approach will work if I want to run on a sweep of node types. But what if I want to run a sweep of model parameters? I'd have to write a python script that would accept input parameters, parse them, etc. That's not very user friendly.

How else could we approach argo where we wouldn't be limited to a conda run command?

For example, I have a script that I want to run that does:

1. Load data and pre-process it
2. Load model and define metrics
3. Evaluate model using model from (2) and data from (1)
4. Save model results

I can currently run this as a single Argo "Step", but what I want is to run these as individual steps so that we get branch and do parameter sweeps, have better visibility and control, etc.

Using the current system I'd have create artifacts to pass objects between the steps and I THINK those would need to be actual files. Each step would have to be a separate script. This is too much overhead. How can we wrap this to make it generalizable and easy for all users?

Value and/or benefit

This would make Argo more useful for ML modelers and simplify a wider list of usecases.

Anything else?

No response

[Adam-D-Lewis]

But what if I want to run a sweep of model parameters?

@kcpevey It'd be nice to include an example if you have one to make this more concrete.

[dharhas]

I'm leaning more and more towards argo not being a good fit for what what we need. I think we need to consider whether flyte or prefect orion etc are better tools.