Test and Step Outputs #23
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kanaka
approved these changes
Dec 4, 2024
jonsmock
force-pushed
the
outputs
branch
2 times, most recently
from
2a5c1c8 to
ca60b3c
Compare
kanaka
reviewed
Dec 5, 2024
The first or primary error is added to the :error key. Additional errors are then appended to the list of :additionalErrors.
jonsmock
force-pushed
the
outputs
branch
4 times, most recently
from
b00a403 to
6f0b02f
Compare
|
@kanaka ok, I think this is ready now, if you want to take another look (just giving you the option for re-review, if you choose/have time). This ensures outputs are captured even on failure, and it starts laying the groundwork for recording multiple errors when they occur (only shown in results-file right now). |
kanaka
approved these changes
Feb 20, 2025
Allow users to define 'outputs' at the step and test levels, which is a map (like 'env') that is interpolated last, after the 'run' command is executed and checked by 'expect' conditions. Having 'outputs' facilitates cleaner reuse of computed values and comparison of values across containers without resorting to writing temp files and/or using ':host'. 'outputs' are always interpolated, even if the test/step has failed, to allow users to inspect/react to issues (ex: using stderr) in subsequent tests/steps. 'outputs' would only be skipped if the step 'if' is false. Besides the addition of the new 'outputs' key, we allow adding 'id' to steps and add two new contexts, 'tests' and 'steps', which represent the previously completed tests and steps respectively. The 'tests' context and the 'depends' key are analogous to 'needs' in GitHub Actions; however, we currently do not check that a user expression that references a test with 'tests' also includes that test in their 'depends' graph. If a user references a non-existent test or non-existent output, they get null (not an exception). Due to our "in place" interpolation of tests and steps, we need to clear uninterpolated 'outputs' in the case the test/step was skipped, so that future tests/steps don't observe uninterpolated strings as 'outputs'. Future refactoring of how tests/steps are interpolated will hopefully make this unnecessary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds off of #22 (to maintain the
./test/runexamplescounts). Only the last commit is new.In short, test/step 'outputs' work like GHA's job-level 'outputs'. I chose not to implement something analogous to GHA's
GITHUB_OUTPUT. Instead we use the same mechanism for both tests and steps.The goal of outputs is to decrease the pain around comparing values within a test (start value vs. end value, or value in one container vs. value in another container) and around capturing/naming computed values to use later (either from earlier step or earlier test).
The new expression contexts are called 'tests' (not 'needs' or 'depends') and 'steps'. I chose to not attempt to check that user-defined expressions that reference the 'tests' context actually refer to a test id in their 'depends' graph (either the immediately named shallow graph or the full resolved graph). Open to discussion here. Right now the user will get a null value if they reference something that doesn't exist, and it's up to them to ensure the right test is depended upon and runs first.
The implementation of this feature is not fantastic, but I think it's really just exposing that the previous
pending->/related work is still not right. I've been working on refactoring this into a form that, instead of interpolating tests and steps "in place" inexecute-stepandrun-test, creates a new binding for the interpolated result and leaves the user's input test or step untouched. I think I'm getting closer, but it has held up the PR for this feature for a couple weeks now. If it is ok, I propose reviewing the 'outputs' feature on its own and that it doesn't make this code significantly worse, and I will continue to explore better ways to interpolate things while handling exceptions etc.