Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
54aa165
add tests from Yi-Cheng
Future-Outlier Nov 27, 2024
9ed6b6e
helped by Kevin and Yi-Cheng
Future-Outlier Nov 27, 2024
4b4f6bd
lint
Future-Outlier Nov 27, 2024
dd774cb
nit
Future-Outlier Nov 28, 2024
0bb8e91
add comments
Future-Outlier Dec 13, 2024
25fea29
add comments and better solution for backward compativle
Future-Outlier Dec 17, 2024
4e24e91
better comments
Future-Outlier Dec 17, 2024
8d1d0e4
DeckStatus
Future-Outlier Dec 18, 2024
31853bb
rename GetDeckStatus
Future-Outlier Dec 18, 2024
4068043
comments
Future-Outlier Dec 18, 2024
65b6efe
lint
Future-Outlier Jan 2, 2025
137579f
fix
Future-Outlier Jan 9, 2025
04f7fbc
Merge branch 'master' into streaming-deck-v2
Future-Outlier Jan 9, 2025
aa56d64
Merge branch 'master' into streaming-deck-v2
Future-Outlier Jan 13, 2025
a16851f
use BoolValue as IDL, suggested by Eduardo
Future-Outlier Jan 13, 2025
7314455
change commennts
Future-Outlier Jan 13, 2025
19498f5
update
Future-Outlier Jan 13, 2025
74f595f
fix
Future-Outlier Jan 13, 2025
3bd3336
fix
Future-Outlier Jan 14, 2025
f6d8493
Merge branch 'master' into streaming-deck-v2
Future-Outlier Jan 14, 2025
4b56e52
fix
Future-Outlier Jan 14, 2025
db4b19e
remove unused ogic
Future-Outlier Jan 14, 2025
2737251
Update flyteidl/protos/flyteidl/core/tasks.proto
Future-Outlier Jan 16, 2025
564dc5f
Update flyteidl/protos/flyteidl/core/tasks.proto
Future-Outlier Jan 16, 2025
69ba94e
Merge remote-tracking branch 'origin' into streaming-deck-v2
eapolinario Jan 16, 2025
c992eae
Merge branch 'master' into streaming-deck-v2
Future-Outlier Jan 17, 2025
0b91b5c
Update by Kevin's advice
Future-Outlier Jan 17, 2025
1d18265
update
Future-Outlier Jan 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions flyteadmin/pkg/repositories/transformers/node_execution.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ func addNodeRunningState(request *admin.NodeExecutionEventRequest, nodeExecution
"failed to marshal occurredAt into a timestamp proto with error: %v", err)
}
closure.StartedAt = startedAtProto
closure.DeckUri = request.GetEvent().GetDeckUri()
return nil
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ var childExecutionID = &core.WorkflowExecutionIdentifier{
const dynamicWorkflowClosureRef = "s3://bucket/admin/metadata/workflow"

const testInputURI = "fake://bucket/inputs.pb"
const DeckURI = "fake://bucket/deck.html"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test here showing that the deck uri also shows up in the case of failed executions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


var testInputs = &core.LiteralMap{
Literals: map[string]*core.Literal{
Expand All @@ -65,6 +66,7 @@ func TestAddRunningState(t *testing.T) {
Event: &event.NodeExecutionEvent{
Phase: core.NodeExecution_RUNNING,
OccurredAt: startedAtProto,
DeckUri: DeckURI,
},
}
nodeExecutionModel := models.NodeExecution{}
Expand All @@ -73,6 +75,7 @@ func TestAddRunningState(t *testing.T) {
assert.Nil(t, err)
assert.Equal(t, startedAt, *nodeExecutionModel.StartedAt)
assert.True(t, proto.Equal(startedAtProto, closure.GetStartedAt()))
assert.Equal(t, DeckURI, closure.GetDeckUri())
}

func TestAddTerminalState_OutputURI(t *testing.T) {
Expand All @@ -84,6 +87,7 @@ func TestAddTerminalState_OutputURI(t *testing.T) {
OutputUri: outputURI,
},
OccurredAt: occurredAtProto,
DeckUri: DeckURI,
},
}
startedAt := occurredAt.Add(-time.Minute)
Expand All @@ -99,6 +103,7 @@ func TestAddTerminalState_OutputURI(t *testing.T) {
assert.Nil(t, err)
assert.EqualValues(t, outputURI, closure.GetOutputUri())
assert.Equal(t, time.Minute, nodeExecutionModel.Duration)
assert.Equal(t, DeckURI, closure.GetDeckUri())
}

func TestAddTerminalState_OutputData(t *testing.T) {
Expand Down
8 changes: 4 additions & 4 deletions flyteidl/clients/go/assets/admin.swagger.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 13 additions & 9 deletions flyteidl/gen/pb-es/flyteidl/core/tasks_pb.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading