Skip to content

Commit c5b787a

Browse files
droctothorpezazulamCarterFendleyedmondopboarder7395
authored
feat(backend): implement subdag output resolution (#11196)
* fix(backend): implement subdag output resolution Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> Co-authored-by: CarterFendley <carter.fendley@gmail.com> * Add support for subdags of subdags Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> Co-authored-by: CarterFendley <carter.fendley@gmail.com> * handle edge case Signed-off-by: zazulam <m.zazula@gmail.com> Co-authored-by: droctothorpe <mythicalsunlight@gmail.com> * Handle artifact outputs as well Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> Co-authored-by: CarterFendley <carter.fendley@gmail.com> Co-authored-by: edmondop <edmondo.porcu@gmail.com> * Simplify parameter handling logic Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> * Begin decomposition Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> * Add support for multiple artifacts and params Signed-off-by: Tyler Kalbach <tkalbach@athenahealth.com> * Implement large tests for subdagio Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> Co-authored-by: CarterFendley <carter.fendley@gmail.com> * Address PR comments & handle oneof Signed-off-by: zazulam <m.zazula@gmail.com> Co-authored-by: droctothorpe <mythicalsunlight@gmail.com> * update backend metadata client test Signed-off-by: zazulam <m.zazula@gmail.com> * Enable nested pipeline IO large tests in CI Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: CarterFendley <carter.fendley@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> * Execute narrow lookup before broad task lookup Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> Co-authored-by: CarterFendley <carter.fendley@gmail.com> * reimplement getDAGTasks to address edge cases Signed-off-by: zazulam <m.zazula@gmail.com> Co-authored-by: droctothorpe <mythicalsunlight@gmail.com> * handle parallelfor in getDAGTask Signed-off-by: zazulam <m.zazula@gmail.com> Co-authored-by: droctothorpe <mythicalsunlight@gmail.com> * Check if iterationIndex is nil Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> * check iteration_count for parallel tasks Signed-off-by: zazulam <m.zazula@gmail.com> Co-authored-by: droctothorpe <mythicalsunlight@gmail.com> * update comment & disable v(4) logs Signed-off-by: zazulam <m.zazula@gmail.com> * Address Chen's feedback Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Co-authored-by: zazulam <m.zazula@gmail.com> * Clarify comment Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> --------- Signed-off-by: droctothorpe <mythicalsunlight@gmail.com> Signed-off-by: zazulam <m.zazula@gmail.com> Signed-off-by: Tyler Kalbach <tkalbach@athenahealth.com> Co-authored-by: zazulam <m.zazula@gmail.com> Co-authored-by: CarterFendley <carter.fendley@gmail.com> Co-authored-by: edmondop <edmondo.porcu@gmail.com> Co-authored-by: Tyler Kalbach <tkalbach@athenahealth.com>
1 parent f2fead5 commit c5b787a

17 files changed

+945
-108
lines changed

backend/src/v2/component/launcher_v2.go

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,17 @@ func (l *LauncherV2) Execute(ctx context.Context) (err error) {
148148
}
149149
}
150150
glog.Infof("publish success.")
151+
// At the end of the current task, we check the statuses of all tasks in
152+
// the current DAG and update the DAG's status accordingly.
153+
dag, err := l.metadataClient.GetDAG(ctx, execution.GetExecution().CustomProperties["parent_dag_id"].GetIntValue())
154+
if err != nil {
155+
glog.Errorf("DAG Status Update: failed to get DAG: %s", err.Error())
156+
}
157+
pipeline, _ := l.metadataClient.GetPipelineFromExecution(ctx, execution.GetID())
158+
err = l.metadataClient.UpdateDAGExecutionsState(ctx, dag, pipeline)
159+
if err != nil {
160+
glog.Errorf("failed to update DAG state: %s", err.Error())
161+
}
151162
}()
152163
executedStartedTime := time.Now().Unix()
153164
execution, err = l.prePublish(ctx)

0 commit comments

Comments
 (0)