ci: deal better with hanging tests #1624
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When two jobs upload artifacts of the same name, a kind of merge happens, and if they write files with the same name, the last one wins. That is undesirable for the artifact that contains failed tests, so let's use different artifact names per job for those. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `workspace` attribute does not exist in the `runner` context, but of the `github` context, so let's use the latter. For details, see: https://docs.github.com/en/actions/learn-github-actions/contexts#github-context Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When tests hang, we do not see anything in the output, and the job just times out after 6 hours (!). To add insult to injury, the timeout prevents the `failed-tests-windows` artifact from being uploaded that is supposed to help with diagnosing the issue better. So let's first establish a generous timeout of 20 minutes for the `win test` matrix jobs (they typically take less than 10 minutes), and then handle the timed-out tests in addition to the failed ones, using the presence of a `.out` file in `t/test-results/` combined with the absence of a `.exit` file as a strong indicator that the corresponding test timed out. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Exactly like we just did with the `win test` jobs, let's establish timeouts for the other jobs that run Git's test suite (giving the ASAN and macOS jobs 60 minutes, all other jobs 20 minutes to finish, which is generous), and detect timed-out tests and handle them the same as failed ones. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I frequently run into CI that times out due to hanging tests. This is an attempt to help with that. This patch series starts out with two slightly-related fixes, then establishes the paradigm with the
win testmatrix, and then applies the same paradigm to the rest of the jobs that run Git's test suite.