Weblogic Operator-3.3.8 rolling restart not happening when the pod is not in Ready State

[jakkoo]

We use Weblogic operator 3.3.8 with FMW 12.2.1.4. In our environment, we have setup the Healthcheck through Weblogic ReadyApp framework. Due to the health check, sometimes pod status will be changed to NotReady or 1/2.

When the pod is not in Ready State, the domain rolling start is not working as expected. Example Assume a weblogic domain with 1 admin pod and 3 managedservers/pods.
Admin ServerPod in Ready state
MS1 pod in Ready State
MS2 pod in NotReady State [NotReady because of Readiness Failure]
MS3 pod in Ready State

In the above condition, if we update the Weblogic DomainResource with a new image then the rolling restart is kicked off.
Until MS1 rolling restart works fine. When it reaches MS2 pod, the rolling restart is not happening and it get stuck in MS2 pod.

In weblogic operator logs able to see every 5 mins, the rolling restart of MS2 pod is attempted but not happening.

Expectation --Rolling Restart of Pod should occur irrespective of POD status or even when the pod is not in Ready State.

[xiancao]

What is your maxUnavailable set? The default value for maxUnavailable is 1. If one server is Unready and unavailable, the operator can't shut down other servers to perform rolling restart. Can you increase the value of maxUnavailable and try again?

[jakkoo]

HiIf I set the maxUnavailable to 2 then two servers will be restarted at same time which we don’t want.Here context is when we patch the new image to the domain resource we expect it to restart the pod irrespective of it’s Ready state.Regards VSOn Apr 25, 2023, at 6:41 PM, Xian Cao ***@***.***> wrote: What is your maxUnavailable set? The default value for maxUnavailable is 1. If one server is Unready and unavailable, the operator can't shut down other servers to perform rolling restart. Can you increase the value of maxUnavailable and try again? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[jakkoo]

Even the maxUnavailable parameter is taking effect only if the POD is in running state. Assume you set maxUnavailable to 2. In this set up, If a POD is in state Init:ImagePullBackOff and if you update the domain resource with new image then also only admin server is getting restarted. The pods which were in "Init:ImagePullBackOff" were not restarted.

[xiancao]

That is by design.

[jakkoo]

HiSo if I understand correctly for domain rolling restart to be success all the pods should be in Running state?If the above statement is true, can it be documented please?Regards VSOn Apr 26, 2023, at 12:20 PM, Xian Cao ***@***.***> wrote: That is by design. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[xiancao]

We support rolling even when the rolling starts while a server is not ready or not yet ready. The maxUnavailable constraint simply must be honored throughout the process.

[jakkoo]

Hi

I have already documented that updating the maxUnavailable from 1 to 2 also doesn’t help. Domain rolling restart doesn’t occur when the pod is not in Ready state. This is our observation even with maxUnavailable value

That’s the reason for raising this bug.

[xiancao]

@rjeberhard ^^^

[rjeberhard]

@jakkoo, I'm following up to see if this is still an issue for you. I apologize that I didn't see the ping from @xiancao.

If this is still an issue, can you please share your domain YAML. I see the discussion about how the setting for maxUnavailable. My expectation is that the operator will wait for not-ready pods to return to ready so that the setting is honored; however, we may be able to do a better job selecting which pod is restarted.