Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split worklfows in DAG #545

Open
ggsdc opened this issue Jul 4, 2024 · 0 comments
Open

Split worklfows in DAG #545

ggsdc opened this issue Jul 4, 2024 · 0 comments
Labels
app Issues relating to the app interface client Issues relating to the client library dags Issues relating to the dags enhancement New feature or request major change Changes that require major changes to the codebase server Issues relating to the server
Milestone

Comments

@ggsdc
Copy link
Member

ggsdc commented Jul 4, 2024

Currently the DAG of a model can perform the following workflows:

  • check input data, solve, check solution data.
  • check input and solution data alone.

With the changes currently in development a new task is added after the solving task to generate a report.

In the future these workflows should be separated into three different DAGs so that we can independently run the following wortkflows:

  • given an execution id: run the input data checks, run the model, run the solution checks.
  • given an execution id: generate the report
  • given an instance id or an execution id: run the input data checks and the solution checks.

That would allow us to run this workflows separately and be able to have a better control over the concurrency of the different tasks.

The activate_dags script should create all the needed DAGs and from cornflow we should have different endpoints to control the different workflows.

In following comments we will add the list of tasks that are needed in order to develop the feature.

@ggsdc ggsdc added enhancement New feature or request server Issues relating to the server client Issues relating to the client library dags Issues relating to the dags app Issues relating to the app interface major change Changes that require major changes to the codebase labels Jul 4, 2024
@ggsdc ggsdc added this to the v2.0.0 milestone Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
app Issues relating to the app interface client Issues relating to the client library dags Issues relating to the dags enhancement New feature or request major change Changes that require major changes to the codebase server Issues relating to the server
Projects
None yet
Development

No branches or pull requests

1 participant