Add proposal for temporary preservation of Failed machines for diagnostics
#1031
Conversation
gardener-robot-ci-2
added
the
reviewed/ok-to-test
thiyyakat
force-pushed
the
proposal/failed-machine-preserve
branch
from
20d8f8f to
9fbdb30
Compare
gardener-robot
added
the
size/m
| * There is a configurable limit to the number of `Failed` machines that can be preserved | ||
| * There is a configurable limit to the duration for which such machines are preserved | ||
| * Users can specify which healthy machines they would like to preserve in case of failure | ||
| * Users can request MCM to delete a preserved `Failed` machine, even before the timeout expires |
There was a problem hiding this comment.
| * Users can request MCM to delete a preserved `Failed` machine, even before the timeout expires | |
| * Users can request MCM to release a preserved `Failed` machine, even before the timeout expires, so that MCM can transition the machine to `Terminating` phase and trigger its deletion. |
| * Users can specify which healthy machines they would like to preserve in case of failure | ||
| * Users can request MCM to delete a preserved `Failed` machine, even before the timeout expires | ||
|
|
||
| ## Solution Design |
There was a problem hiding this comment.
| ## Solution Design | |
| ## Proposal |
| ``` | ||
| * Since gardener worker pool can correspond to `1..N` MachineDeployments depending on number of zones, `failedMachinePreserveMax` will be distributed across N machine deployments. | ||
| * `failedMachinePreserveMax` must be chosen such that it can be appropriately distributed across the MachineDeployments. | ||
| 2. Allow user/operator to explicitly request for preservation of a machine if it moves to `Failed` phase with the use of an annotation : `node.machine.sapcloud.io/preserve-when-failed=true`. |
There was a problem hiding this comment.
Allow user/operator to explicitly request for preservation of a machine if it moves to
Failedphase with the use of an annotation
If the machine already moves to FAILED state and if there is still capacity to preserve MCM will automatically preserve it and if there is no capacity to reserve then it will swiftly move it to termination and trigger its deletion, giving user no chance to influence. I think you meant provide a user an option to preserve specific machines which are not yet in FAILED state, right?
There was a problem hiding this comment.
Yes. Will reword and disambiguate.
| * For a machine thus annotated, MCM will move it to `Terminating` phase even if `failedMachinePreserveTimeout` has not expired. | ||
| 3. If an un-annotated machine moves to `Failed` phase, and the `failedMachinePreserveMax` has not been reached, MCM will auto-preserve this machine. | ||
| 4. MCM will be modified to introduce a new stage in the `Failed` phase: `machineutils.PreserveFailed`, and a failed machine that is preserved by MCM will be transitioned to this stage after moving to `Failed`. | ||
| * In this new stage, pods can be evicted and scheduled on other healthy machines, and the user/operator can wait for the corresponding VM to potentially recover. If the machine moves to `Running` phase on recovery, new pods can be scheduled on it. It is yet to be determined whether this feature will be required. |
There was a problem hiding this comment.
This needs to be reworded and brought out as this is an open question.
Open point:
Should MCM provide an ability to the consumer to trigger drain on the preserved FAILED machine?
| (Running + Requested) | ||
| ├── [Machine fails + capacity available] → (PreserveFailed) | ||
| ├── [Machine fails + no capacity] → Failed → Terminating | ||
| └── [User removes `node.machine.sapcloud.io/preserve-when-failed=true`] → (Running) |
There was a problem hiding this comment.
not clear why User removes node.machine.sapcloud.io/preserve-when-failed=true] → (Running) is here
There was a problem hiding this comment.
For completeness. If the annotation is removed from a healthy machine, the state transitions to (Running) from (Running+Requested), making it a candidate for auto-preservation by MCM on failure, and also allowing users to re-annotate.
| 2. Machine fails later | ||
| 3. MCM preserves the machine (if capacity allows) | ||
| 4. Operator analyzes the failed VM | ||
| 5. Operator releases the failed machine by setting `node.machine.sapcloud.io/preserve-when-failed=false` on the node object |
There was a problem hiding this comment.
This case only talks about Operator suspects a machine might fail and wants to ensure preservation for analysis.
Should it also include explicit release of the preserved-failed machine? Is that not covered in Use Case 4 (early release)?
| Currently, the Machine Controller Manager(MCM) moves Machines with errors to the `Unknown` phase, and after the configured `machineHealthTimeout`, to the `Failed` phase. | ||
| `Failed` machines are swiftly moved to the `Terminating` phase during which the node is drained and the `Machine` object is deleted. This rapid cleanup prevents SRE/operators/support from conducting an analysis on the VM and makes finding root cause of failure more difficult. |
There was a problem hiding this comment.
TLDR: I would not restrict this feature to failing node.
Eg. it can happen that we detect problems with essentially all pods on a node but that this node doesn't report any condition failures (aka the node/machine will not be in failed state).
From an SRE perspective, we want to be as available as possible. Thus, in these kind of cases, we would cordon / drain the pods (except daemonSets) and start investigating the node. Furthermore, since expertise is spread around the globe, we sometimes need to keep a node in a cordoned state for 24-28 hours in order to investigate the root cause with the right area's expert. However, if a node is cordoned with not workload on it, it has very high chances to be scheduled for scale down by CA first.
Thus, this feature should also work for non failing nodes in order to cover all cases.
|
|
||
| This document proposes enhancing MCM, such that: | ||
| * VMs of `Failed` machines are retained temporarily for analysis | ||
| * There is a configurable limit to the number of `Failed` machines that can be preserved |
There was a problem hiding this comment.
FYI, it's rare that many nodes need to be investigated. Since configuration is pretty standard across all node of a same worker group, investigating 1 node is usually enough. That being said, I don't have anything against this protection!
| ``` | ||
| machineControllerManager: | ||
| failedMachinePreserveMax: 2 | ||
| failedMachinePreserveTimeout: 3h |
There was a problem hiding this comment.
I'd put the default of that to 48 or 72 hours as a default. We require expertise from around the world. We also have to include weekends. Since this is going to be a setting shoot-wide, setting it to 3h would in many cases render this feature useless since we can't really change settings in the shoot yaml without the shoot-owner approval.
That being said, if the shoot-owner chooses to set a low amount like 3h on purpose, well then they made the choice to limit support on problematic nodes.
| ``` | ||
| * Since gardener worker pool can correspond to `1..N` MachineDeployments depending on number of zones, `failedMachinePreserveMax` will be distributed across N machine deployments. | ||
| * `failedMachinePreserveMax` must be chosen such that it can be appropriately distributed across the MachineDeployments. | ||
| 2. Allow user/operator to explicitly request for preservation of a specific machine with the use of an annotation : `node.machine.sapcloud.io/preserve-when-failed=true`, such that, if it moves to `Failed` phase, the machine is preserved by MCM, provided there is capacity. |
There was a problem hiding this comment.
Making the annotation be available on both machine and node object is important IMHO, so that any shoot-operator can investigate a failing node by themselves (self troubleshoot)
| * Since gardener worker pool can correspond to `1..N` MachineDeployments depending on number of zones, `failedMachinePreserveMax` will be distributed across N machine deployments. | ||
| * `failedMachinePreserveMax` must be chosen such that it can be appropriately distributed across the MachineDeployments. | ||
| 2. Allow user/operator to explicitly request for preservation of a specific machine with the use of an annotation : `node.machine.sapcloud.io/preserve-when-failed=true`, such that, if it moves to `Failed` phase, the machine is preserved by MCM, provided there is capacity. | ||
| 3. MCM will be modified to introduce a new stage in the `Failed` phase: `machineutils.PreserveFailed`, and a failed machine that is preserved by MCM will be transitioned to this stage after moving to `Failed`. |
There was a problem hiding this comment.
Why not call the status Preserving instead of PreserveFailed (aka as I said above, this would be useful for machines / nodes not detected as failed too)
| ### Use Case 2: Automatic Preservation | ||
| **Scenario:** Machine fails unexpectedly, no prior annotation. | ||
| #### Steps: | ||
| 1. Machine transitions to `Failed` phase | ||
| 2. If `failedMachinePreserveMax` is not breached, machine moved to `PreserveFailed` phase by MCM | ||
| 3. After `failedMachinePreserveTimeout`, machine is terminated by MCM |
There was a problem hiding this comment.
This can be risky of having many nodes in PreserveFailed mainly if we use a relatively high failedMachinePreserveTimeout, even with failedMachinePreserveMax 'cause then a failing worker pool could make multiple nodes getting in PreserveFailed. And depending on how MCM chooses which node to keep / replace in PreserveFailed, could have undesired side effects.
Maybe make that available as an option in the shoot yaml (but defaults to false)?
| ### Use Case 3: Capacity Management | ||
| **Scenario:** Multiple machines fail when preservation capacity is full. | ||
| #### Steps: | ||
| 1. Machines M1, M2 already preserved (failedMachinePreserveMax = 2) | ||
| 2. Operator wishes to preserve M3 in case of failure. He increases `failedMachinePreserveMax` to 3, and annotates M3 with `node.machine.sapcloud.io/preserve-when-failed=true`. | ||
| 3. If M3 fails, machine moved to `PreserveFailed` phase by MCM. |
There was a problem hiding this comment.
FYI changing values directly in the shoot YAML is not always a possibility for operators (E.g. no permissions to edit the shoot YAML (but have admin permission in the cluster). That being said, this is also a valid use case
|
|
||
| ## Open Point | ||
|
|
||
| How will MCM provide the user with the option to drain a node when it is in `PreserveFailed` stage? |
There was a problem hiding this comment.
kubectl drain? I don't see this as something that should be done my MCM IMHO. The exact purpose of this feature is to be able to test a node even if it's failing, thus workload is sometimes needed in order to troubleshoot. I'd add a warning instead in the documentation to do the drain yourself if it's required.
| 2. Since gardener worker pool can correspond to 1..N MachineDeployments depending on number of zones, we will need to distribute the `failedMachinePreserveMax` across N machine deployments. | ||
| So, even if there are no failed machines preserved in other zones, the max per zone would still be enforced. Hence, the value of `failedMachinePreserveMax` should be chosen appropriately. |
There was a problem hiding this comment.
This is confusing to me, but I might not understand the details also... Do you mean that failedMachinePreserveMax is global to all machines / machineDeployment but the maximum amount of nodes in a given MD still takes precedence ?
What this PR does / why we need it:
This PR introduces a proposal to support the temporary preservation of
Failedmachines for diagnostic purposes.Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note: