Allow using Watcher when ConfigMaps and Secrets are maintained by external team

[ismarslomic]

Is your feature request related to a problem? Please describe.

I want to use Spring Cloud Kubernetes Configuration Watcher to monitor and call /refresh for shared ConfigMaps and Secrets maintained by centralised platform team in my project.

The problem is that I do not have permissions to add necessary labels:

• spring.cloud.kubernetes.secret / spring.cloud.kubernetes.secret.informer.enabled
• spring.cloud.kubernetes.config / spring.cloud.kubernetes.config.informer.enabled

and annotations

• spring.cloud.kubernetes.secret.apps / spring.cloud.kubernetes.configmap.apps

in order for Watcher to fire the /refresh calls to the apps.

Note! I do have permissions for getting, listing and watching these ConfigMaps and Secrets, but not editing them.

Documentation I have used so far:

• https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/#spring-cloud-kubernetes-configuration-watcher
• https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/#monitoring-configmaps-and-secrets

Describe the solution you'd like
What would really help is that it was possible (as an alternative solution) to explicitly configure the name of ConfigMaps and Secrets in the Watcher deployment, as environment variables, in addition to which apps to refresh. Or, if you have any suggestion on an alternative solution, not requiring changes to the ConfigMap and Secret resources.

[wind57]

I am a bit confused, you do not have proper rights to create any ConfigMap for example?

[ismarslomic]

Yes, I do have permission to create my own ConfigMaps and Secrets. But in this scenario I want to load and refresh an private token exposed in the shared Secret into my Spring Boot app, and to refresh properties when the private token has been rotated. The creation and token rotation is maintained by the platform team

[wind57]

hmm... we need to know when "something" (configmap/secret in our current implementation) has changed and for that, at the moment, we use informers (think ConfigMap/Secret listeners if you want).

I don't know how would we be able to react to something like : "hey, this has changed; we caught such an event => time to call refresh". As such, I miss your point about:

What would really help is that it was possible (as an alternative solution) to explicitly configure the name of ConfigMaps and Secrets in the Watcher deployment, as environment variables, in addition to which apps to refresh

May be you can explain a bit more verbose please?

[ismarslomic]

I have not complete overview of the current implementation of the Watcher, but conceptually you need following 3 metadata information used for filtering:

• which ConfigMaps and Secrets should I monitor (the annotations spring.cloud.kubernetes.secret.informer.enabled and spring.cloud.kubernetes.config.informer.enabled)
• which ConfigMaps and Secrets should I trigger an changed event for (the labels spring.cloud.kubernetes.secret and spring.cloud.kubernetes.config)
• which apps should I inform about the changes (call to the /refresh endpoint, annotations spring.cloud.kubernetes.secret.apps and spring.cloud.kubernetes.configmap.apps)

Currently this information is configured by using labels and annotations on the ConfigMap and Secret, which works well when each deployment/app has their own ConfigMap and Secret and you maintain those yourself. But in scenarios where these resources are maintained externally, I thought it would be possible to configure these filters explicitly in the watcher.

Basically Watcher needs to monitor all changes to ConfigMaps and Secrets in specified namespace (SPRING_CLOUD_KUBERNETES_RELOAD_NAMESPACES_0) and use the configured filters to decide what to do with them.

[wind57]

which ConfigMaps and Secrets should I monitor (the annotations spring.cloud.kubernetes.secret.informer.enabled and spring.cloud.kubernetes.config.informer.enabled)

A bit not correct. We have two ways for you to tell us which configmaps you want us to watch for:

• If you do not specify spring.cloud.kubernetes.reload.enableReloadFiltering (or set it to false), we will watch everything; every single config map in that namespace will be inspected.
• If you do set spring.cloud.kubernetes.reload.enableReloadFiltering=true, we will only watch configmaps that have a label called spring.cloud.kubernetes.config.informer.enabled=true. This acts as a "filter", so that there is less pressure on the watcher, if that makes sense.

Basically Watcher needs to monitor all changes to ConfigMaps and Secrets in specified namespace (SPRING_CLOUD_KUBERNETES_RELOAD_NAMESPACES_0) and use the configured filters to decide what to do with them.

This already happens, per above. Did you try and it did not work, or the documentation confused you?

[ismarslomic]

If you do set spring.cloud.kubernetes.reload.enableReloadFiltering=true, we will only watch configmaps that have a label called spring.cloud.kubernetes.config.informer.enabled=true. This acts as a "filter", so that there is less pressure on the watcher, if that makes sense.

Where is filtering happening, Watcher or in Kubernetes? Reason why I ask is if the filtering is happening in the code of Watcher then you could use the configuration (provided as env variables) in Watcher as filter, in stead of labels and annotations. My solution might not be the best or most well thought out, but any solution not involving changes to shared ConfigMaps and Secrets would solve the need.

This already happens, per above. Did you try and it did not work, or the documentation confused you?

It does work, Im using it currently.

[wind57]

Filtering is happening in the watcher, yes.

In theory, you could pass those configurations to us, via env variables for example; but how would we know when to trigger a refresh?

[wind57]

hmm, actually I think I get you a little.

The changes in the configmaps still happens, so we can catch that event (this is when refresh must be called). Using the variables like: SPRING_CLOUD_KUBERNETES_SECRET_APPS = app-a, app-b, we could fire refresh event, to these apps.

Did I understand you correctly?

[ismarslomic]

Spot on! The Watcher still listens to all changes (within the specified namespace, if configured), but uses configurations provided by env variables to decide what to do further.

Example

1. What to monitor

Env vars as addition to labels spring.cloud.kubernetes.secret.informer.enabled and spring.cloud.kubernetes.config.informer.enabled:

• SPRING_CLOUD_KUBERNETES_SECRET_INFORMERS = secret-a, secret-b
• SPRING_CLOUD_KUBERNETES_CONFIG_INFORMERS = configmap-a, configmap-b

2. What to trigger event for

Env vars as addition to labels spring.cloud.kubernetes.secret and spring.cloud.kubernetes.config:

• SPRING_CLOUD_KUBERNETES_SECRET = secret-a, secret-b
• SPRING_CLOUD_KUBERNETES_CONFIG = configmap-a, configmap-b

3. Destination of trigger events

Env vars as addition to annotations spring.cloud.kubernetes.secret.apps and spring.cloud.kubernetes.configmap.apps:

• SPRING_CLOUD_KUBERNETES_SECRET_APPS = app-a, app-b, app-c
• SPRING_CLOUD_KUBERNETES_CONFIGMAP_APPS = app-a, app-b, app-c

To be honest I dont personally see the need to differentiate between 1 and 2 as I dont see the value of monitoring something you dont trigger any events for. And, since this solution introduces two ways of defining the same filters (by using env vars and labels/annotations) we could define some "merge" principles when both mechanisms are used?

[wind57]

yeah, this sounds like a reasonable enhancement to me, but there are things to think about here, merging is one, backwards compatibility is another... I'll see if I can find some time next week to think more about this.

@ryanjbaxter your input is needed as always :)

[ismarslomic]

Great! Thanks a lot! Im happy to create an working example we could use to demonstrate and test this solution, using latest version of Spring Boot and Spring Cloud Kubernetes libs.

[ryanjbaxter]

I think we would need to make the configuration a bit more robust if we are configure things via properties in the watcher itself.

Something like this would make more sense to me:

spring:
  cloud:
    kubernetes:
      config:
        watcher:
          watch: [{
            namespace: [list of namespaces or * for all namespace]
            configmaps: {
              names: [list of config maps names or * for all config maps]
              apps: [list of apps to refresh]
            }
            secrets: {
              names: [list of secret names or * for all secrets]
              apps: [list of apps to refresh on changes]
            }
         },
         ....
        ]

[wind57]

This is next on my TODO list, but I am heading for a vacation in a week or so. Nevertheless, I will start thinking and working on it. FYI

[ismarslomic]

Thanks for the update @wind57, no hurry from my side! And I agree with your suggestion @ryanjbaxter!

I just wanted to share some feedbacks from other tech leads in my company, after doing a demo earlier today. One important concern they raised were that the push-pattern where the Watcher is triggering a refresh in apps is that there is no guarantee that the actual ConfigMap/Secret value in mounted files has been updated at that point of time (eventual consistency in Kubernetes). Yes, we can increase the refresh delay in the Watcher to some point, and reduce the uncertainty, but we actually don't know if change has been applied.

Their suggestion where to let the Spring Boot app monitor file changes of it's mounted files (for ConfigMaps and Secrets), and trigger a refresh when change event is observed. This would eliminate the need of an centralised Watcher and reduce uncertainty that the refresh was trigged before the actual value has been updated in mounted files in app's container.

I still think that suggested solution in this feature enhancement is valuable for those considering using Watcher, but I'm not sure if we are going to use it in our architecture.

[ryanjbaxter]

I believe when I first wrote this feature that there was a maximum amount of time it could take before the mounted file was updated. Anything under that time was not guaranteed, but I believe I set the default at the max.

But yes monitoring the file system is another solution.

[wind57]

I'm sorry for the long delay, I was caught up with quite a lot at my workplace, but I plan to start looking into this one fairly soon... just FYI