Skip to content

Seamless operator upgrades #967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 13, 2025

Conversation

jswoods
Copy link
Contributor

@jswoods jswoods commented May 1, 2025

Adds support for seamless operator upgrades by:

  • Introducing managedFields on humio pods which list the "humio-operator" has the owner for fields on the humio pods that are using the operator-supplied defaults that are not specified in the HumioCluster CR
  • Splitting the pod hash into two separate annotations. The existing annotation only tracks the non-operator managed fields (those set at the HumioCluster CR) and a new annotation called PodOperatorManagedFieldsHashAnnotation which tracks only the fields set by the operator defaults. Pod replacement logic is expanded to only replace pods when the non-operator managed fields hash annotation changes and log a warning and not restart pods when the managed fields hash annotation changes.

Note that some exceptions exist, which include the main humio version (due to the upgrade logic) as well as some resources that do not typically change, such as default requests/limits, etc. Adding support for these is simple and can be done in this change if we'd like, or we can instead track these only when the time comes where we need to update them.

By far, most of the cause for the humio pod restarts occur when the helper image changes, and this is supported as a managed field as well as any environment variables that are added by the operator as defaults.

Also added helm upgrade tests which by using the new HUMIO_OPERATOR_DEFAULT_HUMIO_HELPER_IMAGE_MANAGED in the test will capture what happens when the operator helper image default changes. This way the test can be run against the current version of the code rather than only against older versions which are "upgraded" to the current. Tests for both are included but we may chose to remove the upgrade-from-an-older-version test for maintainability.

Fixes #246.

@jswoods jswoods force-pushed the jestin/seamless-operator-upgrades-split-annotations branch 3 times, most recently from 07d366e to b74a669 Compare May 7, 2025 03:53
@jswoods jswoods marked this pull request as ready for review May 7, 2025 14:04
@jswoods jswoods requested a review from a team as a code owner May 7, 2025 14:04
@jswoods jswoods force-pushed the jestin/seamless-operator-upgrades-split-annotations branch from d41bd5f to bd03ec8 Compare May 12, 2025 17:01
@jswoods jswoods merged commit 3751f3d into master May 13, 2025
25 checks passed
@jswoods jswoods deleted the jestin/seamless-operator-upgrades-split-annotations branch May 13, 2025 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrading operator kills all pods
3 participants