exec: refactor command execution #486
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a go program hangs forever, the go runtime detects it and terminates it
with the error message:
fatal error: all goroutines are asleep - deadlock!
This happened in the TestProcessTerminatesWithParent, where a child process was
only running `select {}` to wait forever until it gets killed.
Replace select{} with a very long time.Sleep() call in the
TestProcessTerminatesWithParent() to prevent that the testcase fails because the
child process gets terminated because of the described reason.
- When a command is executed, all it's stdout and stderr output is stored in a
buffer in memory. The buffer is used when evaluating the command output and to
print it if the command execution should fail.
This can be very memory intensive when executed tasks generate a lot of
output. To reduce the memory usage:
- only store up to 16KB of the beginning and up to 16KB of the end of the output
per stdout and stderr for the eventual error message when the
command execution fails. The truncated part is marked in the error message.
This should be sufficient context to understand failures.
This is implemented with a modified copy of the prefisuffixsaver from the
os/exec stdlib package.
- The Run() function does not store and return the whole output of a command
on success anymore. Run() is used to execute the commands of baur tasks.
This will reduce the memory usage.
- The new RunCombinedOut() stores the stdout output in a buffer. This
function is used to run baur internal commands that only generate few
output and the output is required for evaluation.
- running commands can now be canceled via a passed context. This is not used by
anything yet. It will be used to specify timeouts for "baur run".
- Cmd now supports to specify an additional io.Writer to stream the stdout and
stderr output of a command. This enables efficient processing of larger
outputs and will be used in a following Pull-Request, to run "git ls-files" to
obtain input file digests.
- when the command output is streamed to the terminal
(baur run --show-task-output), the stderr output is now colored red, that
makes it distinguishable from stdout output
- when --show-task-output is not passed to baur run and the execution of a
command fails, the stderr and stdout output will now be shown after each other
instead of intermixed. This ensures that output of both file descriptors can
not become interlaced and result in bogus output.
…sk-output When "--show-task-output" is passed to "baur run", the command output is now written to stderr instead of stdout. Stderr fits better the output is for diagnostic purposes.
nocive
approved these changes
Jan 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a command is executed, all it stdout and stderr output is stored in a buffer in memory. The buffer is used when evaluating the command output and to print it if the command execution should fail. This can be very memory intensive when executed tasks generate a lot of output. To reduce the memory footprint: - only up to 16KB of the beginning and up to 16KB of the end of the output is stored per stdout and stderr for the eventual error message if the command execution fails. The missing part is marked in the error message. This should be enough to understand failures and their context. This is implemented with a modified copy of the prefisuffixsaver from the os/exec stdlib package. - The Run() function does not store and return the whole output of a command on success anymore. Run() is used to execute the commands of baur tasks. This will reduce the memory usage. - The new RunCombinedOut() stores the stdout output in a buffer. This function is used to run baur internal commands that only generate few output and the output is required for evaluation.
running commands can now be canceled via a passed context. This is not used by anything yet. It will be used to specify timeouts for "baur run".
Cmd now supports to specify an additional io.Writer to stream the stdout and stderr output of a command. This enables efficient processing of larger outputs and will be used in a following Pull-Request, to run "git ls-files" to obtain input file digests.
when the command output is streamed to the terminal (baur run --show-task-output), the stderr output is now colored red, that makes it distinguishable from stdout output
when --show-task-output is not passed to baur run and the execution of a command fails, the stderr and stdout output will now be shown after each other instead of intermixed.
Closes #328