Skip to content

Increase internal-request timeout to match #449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 12, 2025
Merged

Conversation

ralphbean
Copy link
Member

The TIMEOUT value here is the time that the parent release pipeline will wait for the internal request to finish, and the other timeouts are the timeouts applied to the child internalrequest pipeline.

If these are unequal, things don't make sense. We had a case today where an internalrequest pipeline took 20m. It succeeded in the end, but only after the parent release pipeline had given up on it. This can cause a thundering herd problem in the downstream cluster where internal requests pipelines execute, when parent pipelines retry and stack up more downstream internalrequests that end up going unused.

@ralphbean ralphbean requested a review from a team as a code owner June 11, 2025 19:16
happybhati
happybhati previously approved these changes Jun 11, 2025
@ralphbean ralphbean force-pushed the timeout-adjustment branch 2 times, most recently from e8827de to 2756d97 Compare June 11, 2025 19:32
Copy link
Collaborator

@johnbieren johnbieren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lines 35 and 74 say this value too, can you update them?

@happybhati happybhati self-requested a review June 11, 2025 19:49
The TIMEOUT value here is the time that the parent release pipeline will
wait for the internal request to finish, and the other timeouts are the
timeouts applied to the child internalrequest pipeline.

If these are unequal, things don't make sense. We had a case today where
an internalrequest pipeline took 20m. It succeeded in the end, but only
after the parent release pipeline had given up on it. This can cause a
thundering herd problem in the downstream cluster where internal
requests pipelines execute, when parent pipelines retry and stack up
more downstream internalrequests that end up going unused.

Signed-off-by: Ralph Bean <rbean@redhat.com>
@johnbieren johnbieren merged commit 6a1df8c into main Jun 12, 2025
3 checks passed
@johnbieren johnbieren deleted the timeout-adjustment branch June 12, 2025 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants