[Enhancement] Watch for certificates in subdirectories for path specified in "watchDirectories" attribute.

[porwalameet]

Discussed in #107

^{Originally posted by porwalameet February 28, 2023}
Hi All,
Thanks for this wonderful project which helps us monitor certificates on Kubernetes Cluster.

We are using daemonset mode with "watchDirectories" setting. This address most of our use-case, however at times we have certificates mounted onto pod-filesystem directly using CSI Volumes or configMap.
Example:
A certificate is stored in Hashicorp Vault or Azure KeyVault, and using CSI, we can pull these certificates and mount directly on pods.

Here there are no kubernetes secret created and wish to monitor these certificates as well.

However, the files reside on respective node filesystem where pod is run in /var/lib/kubelet/pods/<pod-uid>/volumes/<volume-type>. The pod UID is unique and dynamic uid which is assigned to a running pod and remains till the pod lifetime.
On host Node, it looks something like below:

# pwd
/var/lib/kubelet/pods/c5ce7790-8260-4e30-8f56-8ef6a374f755/volumes
# ls -ltr
total 0
drwxr-xr-x 3 root root  27 Feb 27 10:24 kubernetes.io~downward-api
drwxr-xr-x 5 root root  67 Feb 27 10:24 kubernetes.io~configmap
drwxr-xr-x 7 root root 103 Feb 27 10:24 kubernetes.io~empty-dir
drwxr-xr-x 4 root root  54 Feb 27 10:24 kubernetes.io~projected

So, to address this we can specify watchDirectories as /var/lib/kubelet/pods/*/volumes or just /var/lib/kubelet/pods/, which scans all pod directories and appends to existing list for the files scanned and follow the basic flow of parsing certificates.

As watchDirectories configuration monitors only files within the specified path, is there any work going on to address recursive parsing in subdirectories.

Thanks.

[porwalameet]

If this enhancement is acceptable, I can pick it up and contribute for same.

[npdgm]

Hi @porwalameet

Sorry we didn't get back to you sooner. There have been other requests for this enhancement already, but your issue certainly revived the discussion on how we could implement that. Work is in progress.

Actually there was a good reason why this directory watch option never had recursion. It comes to the fact there is no identification of certificate files based on filenames, such as extensions: .crt, .pem, ... Crawling a large filesystem would instantiate the PEM parser for every single file, consuming memory and cpu resources, and probably taking too much time to answer scrape queries before the cache gets fed. And since we do not maintain the parser code which is from Golang, that's also an open door for regressions in performance or behaviour, should we rely too much on it for scanning content in tens of thousands of files likely found in a /var/lib/kubelet/pods/.
So it was a safeguard against configurations that would make the exporter behave poorly and likely far bellow user expectations.

Ultimately we'll need an optional configuration file to add many options to each file or directory path.
In the meantime this can be implemented in a limited form with CLI arguments. Such as giving a list of file extensions, or whole path globbing.

Let me sync up with my colleague who has been looking at that feature already, and I'll tell you what to expect.

[porwalameet]

Thanks @npdgm. I do agree there will lot of files and parsing might take lot of memory based on pods running on a node. We can filter volumes directory specifically - like /var/lib/kubelet/pods/*/volumes, since such secrets/certificates will be volume mounted, so the scanning target is limited now, we can have such optimizations to narrow it down further. Just a thought.

[porwalameet]

Hi @npdgm , just checking did you hear back anything on this feature/Enhancement.

[arcln]

Hello, I am almost done implementing this into the exporter. Then we'll need to update the helm chart to be compatible with the changes, and then we'll release this feature. You can expect to see it released within 1-2 weeks.