Get previous custom resource version on update

[Crespalvaro]

I might be missing something or I didnt got the concept of how I should work with this. However, I have a custom resource and I want to know what changed against the previous version to execute proper actions.

[buehler]

Hey @Crespalvaro
What do you mean by "what has changed"? Currently the actual changes in the CRD are not visible, you'll receive an update (modified) event if the resource has changed, but you do not receive the actual changes.

[Crespalvaro]

Hello @buehler
I'll try to explain better which is my objective. Rather than get an input with the exact changes on an Update event, I would like to be able to check what changed between current version of the custom resource that triggered the update event and the previous version.

I'm not expecting the operator it self to give an input object with the exact changes between both versions, but if I would be able to access both versions at the same time I could allow my controller to programmatically check the changes.

An example of this is, with kubernetes client, you can get a watcher object through a list operation. This whatcher contains the version when the object was created and following version in each update.

However I'm stuck as I couldn't find exactly how to work with this watcher but I believe that at the end your operator works with them to monitor custom resource events and then forward them to the create, update, not modified, status changed and delete events that you are providing with the operator (if I'm not wrong)

At the end, I don't know if my concep of how to use a controller/crd is wrong. To give you a bit of context, my controller is reading all the config data to install 5 helm charts using flux-controller and helm-controller (very nice controllers btw). Therefore, on a custom resource update I want my controller to know which helm chart has to upgrade.

Thanks a lot for your support

[erin-allison]

@Crespalvaro I think trying to reconcile only certain changes like that is a bit of anti-pattern. The idea behind a controller and the declarative configuration is that whenever you reconcile the resource, you check all components of it to protect against external drift - the current/past state of the resource is irrelevant, you simply calculate what needs to occur to bring the system to the desired state.

For an example, if you have a replica-set specifying to run 8 of a certain pod, and you delete one, the controller that manages that resource type will automatically re-create the missing pod to attain the desired state defined by the resource.

In your case, if you were to only modify the helm charts if/when the associated parts of your custom resources were updated, someone/something could uninstall/modify those charts externally and your controller would never change it back. This could easily end up with a broken state if one of those charts depends on others or other similar situations.

[Crespalvaro]

@erin-allison Got your point. This means that on update event further than just check what changed I should check that every component related with my cr accomplish with it. And, following your example, if you updated the spec to have 9 replicas, you will have to scale up the deployment.
Understand that all this process must be recurrent in notmodified event and update even is kind of just an event to tell the operator to do it in that moment as it's sure that something changed and an action is required. More or less.

[buehler]

Hey @Crespalvaro
I understand what you mean. It is theoretically possible. There are Kubernetes metadata fields that allow you to do such things (more or less). For example, when you apply the CRD with kubectl apply, the last used configuration of the entity is stored in an annotation kubectl.kubernetes.io/last-applied-configuration. It contains the last json that was applied.

However as @erin-allison pointed out, this is not the general idea of the reconciliation loop in the operator pattern. You should always check if the given state in the cluster matches the one that is described by the entity.

Now when I'm thinking about, the differentiation between "Created" and "Updated" is useless, because one should always check the whole state of the entity. When I created the SDK, I thought this difference matters ¯\_(ツ)_/¯

[Crespalvaro]

Hello @buehler
Im not sure last-applied-configuration works as k8s intended it to work but.... Anyways you placed me in the right path, my concept about the subject was wrong. Thanks a lot.

Well I dont this Update is useless. By default notModified event has a requeue of 5 seconds and we are planning to go to production in about a month. Once we are in prod most likely we dont want to trigger this event every 5 seconds therefore as Update method will tell the controller exactly when to perform a check. Even though you still have to check the whole entity on Update, you wont recreate everything, just those components that rlly need an update.

Anyways, thanks a lot.

Just another hint now we are in conversation. Still need to dig into it, but It would be amazing that when building operator config (crds and so on) instead of replacing kustomize files, mergin as git does could be valuable as it would prevent you from losing custom changes. Ill dig into alternatives and If I build something I might send you a pr.

Once again, thanks a lot for your work with the SDK

[buehler]

You're welcome! It is a fun project, and it makes me happy to see that people use it

Regarding the update: well, yes, but it encourages behavior that differentiates between created and updated. This only works as long as the operator is not restarted. When the operator restarts, the local item cache is gone and all entities are throwing "created" events again. So to ensure that the intended way of the operator pattern is used, the difference between created and update should be removed.

Regarding the created yaml's: Currently the mechanism is that "elements that need customization" - like installer directory and dockerfile - are not overwritten. All other elements are overwritten since they are entirely generated by the code in the operator (rbac roles, controller definition, crd's)