Support retrieval of temporary published outputs #146
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Anticipating having this function deligate depending on the output type
This lets us more clearly specify which output type is used in a recipe
trey-stafford
force-pushed
the
110-return-published-outputs
branch
from
6fafb9d to
71bc4dd
Compare
I could have sworn this was the default behavior...?
trey-stafford
force-pushed
the
110-return-published-outputs
branch
from
59bf0fe to
718897b
Compare
For now we can just always assume a recipe with a temporary output has not been published.
trey-stafford
force-pushed
the
110-return-published-outputs
branch
from
227e98a to
9401699
Compare
trey-stafford
commented
Jan 16, 2026
| input: RecipeInput | ||
| output: RecipeOutput = RecipeOutput() | ||
| output: DataOneRecipeOutput | TemporaryRecipeOutput | PvcRecipeOutput = ( | ||
| PvcRecipeOutput() |
Member
Author
There was a problem hiding this comment.
Default to PVC output, which was the only and default behavior prior to this PR. This makes it a little more explicit/clear. But do we want this as the default? Maybe temporary output makes more sense as a default?
trey-stafford
commented
Jan 16, 2026
| # location uses a key that's based on the argo workflow name. We could | ||
| # probably use `hera` to filter for a matching workflow based on the | ||
| # `recipe_config.id`, and check its output, but this will be much easier | ||
| # to implement once we track recipe executions in the database. |
This should be temporary. Need to address the db component to track recipe executions.
trey-stafford
force-pushed
the
110-return-published-outputs
branch
from
b6dc302 to
d102260
Compare
This is required so that swagger knows it lives under a root path when behind ingress/proxy
trey-stafford
force-pushed
the
110-return-published-outputs
branch
from
6338f36 to
882435d
Compare
trey-stafford
commented
Jan 20, 2026
trey-stafford
added a commit
to QGreenland-Net/ogdc-recipes
that referenced
this pull request
Jan 20, 2026
trey-stafford
changed the title
rushirajnenuji
approved these changes
Feb 2, 2026
Member
rushirajnenuji
left a comment
rushirajnenuji
left a comment
There was a problem hiding this comment.
Reviewed and tested this locally, and it looks good. All my tests passed locally. 🚀
One thing that we should add in the future iterations is the workflow ownership checks before retrieving the temp output artifacts generated by the workflow.
We can go ahead and merge this PR, thank you @trey-stafford for adding this support. 🎉🚀
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a new endpoint and CLI for downloading the results of a workflow with a
temporaryoutput type. Resolves #110.See also:
outputcfg perogdc-runner#146 ogdc-recipes#10Prior to this PR, final outputs would be placed in a PVC, and these were not particularly accessible without direct access to the PVC. Now there is a new output type,
temporary, which zips the final output of a recipe and places it Argo's configured artifact storage (s3). The new service endpoint takes a workflow name and returns a pre-signed URL to the object that users can use to download the data.The pre-signed url allows access to the data object for 2 hours, and then the URL expires.
The idea here is that workflows with temporary outputs can be cleaned up after a given amount of time. This works well for recipes that do not need to formally publish outputs (e.g,. a subsetting operation).
Along with the PRs mentioned above, the API landing page should not be accessible at: https://localhost:7443/ogdc/api . Note that you will need to accept a security exception, as this HTTPS endpoint utilizes a self-signed (unverified) cert.