Fix: libpe_status: Use pcmk_monitor_timeout for recurring monitors #3246
Conversation
nrwahl2
force-pushed
the
nrwahl2-rhel14826
branch
from
8251f1a
to
4378bf4
Compare
The executor uses pcmk_monitor_timeout, but the controller considers a recurring monitor to have timed out after its op timeout expires. If pcmk_monitor_timeout is very long (for example, 240 seconds), a stonith stop action can fail. In this situation, the monitor is declared as timed out before the pcmk_monitor_timeout expires, the stop action is requested, and its timer begins counting down. However, the stop action can't begin until after the monitor finishes or pcmk_monitor_timeout expires. This also makes special handling in controld_execd.c unnecessary. pcmk__unpack_action_meta() has already replaced the meta timeout with the pcmk_monitor_timeout. Closes RHEL-14826 (JIRA). Signed-off-by: Reid Wahl <nrwahl@protonmail.com>
nrwahl2
force-pushed
the
nrwahl2-rhel14826
branch
from
4378bf4
to
5eb3d24
Compare
| && (pcmk__str_eq(action_name, PCMK_ACTION_START, pcmk__str_none) | ||
| || pcmk_is_probe(action_name, interval_ms))) { | ||
| && pcmk__str_any_of(action_name, PCMK_ACTION_START, PCMK_ACTION_MONITOR, | ||
| NULL)) { |
There was a problem hiding this comment.
See e44a6d4 commit message -- will the controller think the action configuration changed?
There was a problem hiding this comment.
Thanks for pointing that out. The commit message is long and I missed that while refreshing for this "easy fix"...
Sigh, almost certainly. This is probably going to be a CANTFIX, unless we decide the digest change is acceptable in order to avoid a failed stonith stop action.
If it would make any difference in our decision, I'll double-check whether this can happen with any timed-out recurring monitor (with long pcmk_monitor_timeout) or if it's specific to the first one somehow. If it can happen with any (which it looks like should be possible), I'm surprised no one has hit and reported this.
There was a problem hiding this comment.
That, or try to find a different way to update the controller's expected timeout, which may be considerably more complicated
There was a problem hiding this comment.
We do already have a mechanism in the fencer for updating the controller's expected timeout for fencing actions, so it could be modeled on that, but it would still likely be pretty intrusive.
Doesn't the controller have both timeout and pcmk_monitor_timeout in the graph? Maybe it could just do the override itself.
There was a problem hiding this comment.
Doesn't the controller have both timeout and pcmk_monitor_timeout in the graph? Maybe it could just do the override itself.
IIRC we strip out pcmk_* options from the transition graph. I suspect a fairly straightforward fix would be to add pcmk_monitor_timeout to the graph when needed, as a special XML attribute rather than with the rest of the resource parameters (to avoid breaking the hash). The controller can then pull it out and use it instead of the usual timeout.
The executor uses
pcmk_monitor_timeout, but the controller considers a recurring monitor to have timed out after its op timeout expires. Ifpcmk_monitor_timeoutis very long (for example, 240 seconds), a stonith stop action can fail. In this situation, the monitor is declared as timed out before thepcmk_monitor_timeoutexpires, the stop action is requested, and its timer begins counting down. However, the stop action can't begin until after the monitor finishes orpcmk_monitor_timeoutexpires.This also makes special handling in
controld_execd.cunnecessary.pcmk__unpack_action_meta()has already replaced the meta timeout with thepcmk_monitor_timeout.Closes RHEL-14826 (JIRA).