Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(auto-cancel): server-side logic for auto canceling obsolete builds #911

Merged
merged 13 commits into from
Oct 20, 2023

Conversation

ecrupper
Copy link
Contributor

@ecrupper ecrupper commented Jul 18, 2023

Depends on go-vela/types#299

@codecov
Copy link

codecov bot commented Oct 9, 2023

Codecov Report

Merging #911 (f4f3d36) into main (d90fdb7) will decrease coverage by 0.72%.
The diff coverage is 14.79%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #911      +/-   ##
==========================================
- Coverage   71.10%   70.39%   -0.72%     
==========================================
  Files         313      315       +2     
  Lines       13028    13195     +167     
==========================================
+ Hits         9263     9288      +25     
- Misses       3299     3440     +141     
- Partials      466      467       +1     
Files Coverage Δ
database/resource.go 72.03% <ø> (ø)
database/validate.go 100.00% <ø> (ø)
database/build/list_pending_running_repo.go 89.28% <89.28%> (ø)
api/build/cancel.go 0.00% <0.00%> (ø)
api/build/auto_cancel.go 0.00% <0.00%> (ø)

@ecrupper ecrupper marked this pull request as ready for review October 9, 2023 20:57
@ecrupper ecrupper requested a review from a team as a code owner October 9, 2023 20:57
api/build/cancel.go Show resolved Hide resolved
api/build/cancel.go Show resolved Hide resolved
timhuynh94
timhuynh94 previously approved these changes Oct 12, 2023

req, err := http.NewRequestWithContext(context.Background(), "DELETE", u, nil)
if err != nil {
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are your thoughts on attempting the other cancels in the event of an error on 1 of them? maybe track an error in the outer scope and check that at the end? idk

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean like if there are multiple running builds to auto-cancel, and one of those attempts fails?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i was looking at the loop over executors

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one build will match, so there should only be one attempt at cancellation.

But we do loop over pending/running builds: https://github.com/go-vela/server/pull/911/files#diff-2e48a4018b70aac91235c0a2f5ec50d7ae186de39e3bd603d7e0fcfe6fd5ab07R25-R65.

I was unsure as to whether or not to immediately error out upon failing one of those (provided they match all the other criteria). What do you think? If this policy was implemented, there really should only be one build at a time that gets the boot. Each new build will supersede the older build, which superseded a previous build, and so on.

All's that to say, even though there are a couple loops here, we're really just performing one operation overall — I think... 😅

api/build/auto_cancel.go Outdated Show resolved Hide resolved
Copy link
Member

@wass3rw3rk wass3rw3rk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the current implementation has the potential to take some time before the current build makes it onto the queue dependent on how many builds need to be canceled and how many containers each build spun up.

It waits until all builds are canceled before pushing the current build to the queue. Should we send current build to queue and, in best effort, send off cancellation attempts to the builds that we need to cancel?

Copy link
Member

@wass3rw3rk wass3rw3rk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, if it fails canceling any one of the builds it will not attempt to cancel any of the other builds it needs to cancel, i believe? i could be wrong - but if not, i think we want to continue attempting to cancel the other builds.

@ecrupper ecrupper merged commit b51d5a2 into main Oct 20, 2023
12 of 13 checks passed
@ecrupper ecrupper deleted the hackathon-auto-cancel branch October 20, 2023 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants