How to UpdateStatus and requeue "Reconcile" events until a desired state is reached?

[Karthik2893]

I have a CRD which defines a resource (say X). CRUD operations on X are handled by a different service which returns a "requestId" when a resource creation is requested. Following are the sequence of operations I am currently doing in my controller:

When a custom-resource is created, I am calling the service to create the resource.
Updating the "Status" to reflect the "requestId" and the "state" of X.
If the "state" != "Succeeded", requeue "Reconcile" event with a delay.
If state == "succeeded" , return null.
In my observation, whenever I add "Step 2", the "Reconcile" action/method is not being invoked on the controller.

Am I doing something of an anti-pattern here?

Some due diligence (incase it might help anyone looking in anyway):
I have a very basic understanding of reactive streams, so please bear with me. My understanding from this code is that we are grouping events for each individual resource and by invoking "Switch", only projecting the recent observables? Could this be causing this by any chance?

dotnet-operator-sdk/src/KubeOps/Operator/Controller/EventQueue.cs

Lines 44 to 55 in 3f42bbc

	Events = _localEvents
	.Merge(watcherEvents)
	.Where(EventRetryCountIsLessThanMax)
	.GroupBy(e => e.Resource.Uid())
	.Select(
	group => group
	.Select(ProcessDelay)
	.Switch())
	.Merge()
	.Select(UpdateResourceData)
	.Merge()
	.Where(EventTypeIsNotFinalizerModified);

[buehler]

Hey @Karthik2893

In theory, you should be able to check your resource (the state of it), and then either return a requeue request or null. If null is returned, the resource does not get reconciled right now (only when the watcher terminates the connection).

The code you mention groups all events by resource id. The "ProcessDelay" method returns a new observable with a delayed start (if any) and the switch changes to that observable instead. This should not impact your usecase.

It is not an anti-pattern (IMO). You create an instance of a CRD, which then creates some external resource. This is totally fine.

sequenceDiagram
actor u as User
participant a as API
participant o as Operator
participant e as External

u ->> a: Create CRD instance
a ->> o: Instance Created
o ->> e: Create

loop Check Status
    o ->> e: Status?
    
    alt Successful
        o ->> o: Resolve reconcile with "null"
    else
        o ->> o: Resolve reconcile with re-queue
    end
end

Loading

Is this diagram showing your use-case or did I get something wrong? If yes, this should be possible.

[Karthik2893]

Thanks @buehler for taking the time out and responding to my question. The use-case I intended to describe above is a little different. Following is the updated picture that reflects my use-case:

In the outer loop, everytime I retrieve the Status from the external service, I am doing the following:

1. Update the "Status" sub-resource of my CRD instance (via UpdateStatus on KubeOps.KubernetesClient). I want the Status on the CRD to always reflect the most recent state of the resource.
   The resource creation goes through several steps before it ends up in a "Succeeded" or "Failed" state and for our use-case, it is helpful to reflect the most recent state on the CRD instance.
2. If resource creation is successful (or Failed), resolve reconcile with "null".
3. Else, resolve reconcile with re-queue. (This means the resource creation is not yet complete and we want to wait and check the status after the specified delay)

Following are my observations:

1. When the "Status" on the CRD instance is updated and I resolve reconcile with a "Requeue", I am noticing the "StatusModifiedAsync" method is invoked on the controller, but the requeue never happens.
2. If I don't update the "Status" on the CRD instance and I resolve reconcile with a "Requeue", then the requeue works as expected.

I am assuming that the "Status" on the CRD instance can be updated to reflect the current state of the resource and we should have the provision to resolve reconcile with "Requeue" as long as the resource is not in one of the desired terminal states. Is this an anti pattern?

I linked the below snippet because I suspected that the "Switch" might be causing this behavior. Going strictly by the definition, "Switch" produces values only from the most recent observable sequence. So, I was thinking the "Requeue" events are being ignored whenever the "StatusModified" events are posted. (I have never used Rx before, so my understanding is very theoretical, please ignore if I got this wrong ).
I replaced "Switch" with "Merge" and then I observed that "Requeue" events are being handled after the "StatusModified" events. (I could very well be introducing unintended side-effects with this change and I am by no means suggesting that this is a solution. Just wanted to bring it to your notice incase it might help).

dotnet-operator-sdk/src/KubeOps/Operator/Controller/EventQueue.cs

Lines 44 to 55 in 3f42bbc

	Events = _localEvents
	.Merge(watcherEvents)
	.Where(EventRetryCountIsLessThanMax)
	.GroupBy(e => e.Resource.Uid())
	.Select(
	group => group
	.Select(ProcessDelay)
	.Switch())
	.Merge()
	.Select(UpdateResourceData)
	.Merge()
	.Where(EventTypeIsNotFinalizerModified);

Thank you.

[buehler]

Alright, I understand the use-case and no, it is not an anti-pattern. This behavior can be seen in many operators. This is a bug.

Since the status update does trigger and the reconcile doesn't, the switch/merge magic you point out may be the issue. The switch is there to prevent duplicate events, in case of status only updates however, they should fire.

[Karthik2893]

Hi @buehler , just a gentle reminder to check out my above response from last week regarding. It would help us get some clarity on our implementation and I'd really appreciate your input. Let me know if you have any questions.

[buehler]

This is a bug in the SDK.
I created an issue for it: #554