External services depend on Alloy chart's improper default cluster-wide secret read permissions

[kallayj]

See grafana/alloy#2224.

This change introduced a dependency on the Alloy chart being installed with read permission to the whole cluster's secrets, even in the default case where the secret is being created by the k8s-monitoring chart in its own namespace. If the issue above is fixed to require explicit opt-in to exposing secrets, as I think it must, that implementation will have to be reconsidered.

[petewall]

Yeah, I never loved that it uses cluster-wide secrets reading.

A few things that can be done to address this:
Short term, we can turn it off by modifying the RBAC rules. Then, if you want to use a secret in a different namespace, we either do our best to detect that and warn you at deployment time, or put in the troubleshooting doc what to do when we see those errors in the logs.

Long term, the k8s-monitoring chart takes on the role of building RBAC (or at least the secrets portion) based on what the user has specified.

[kallayj]

When I opened the issue my (less-than-fully-informed) impression was that the use of remote.kubernetes.secret for the credentials between Alloy and the external services was to bypass the namespace sandbox that is imposed on the standard mechanism for mounting secrets as environment variables. Since the feature intent was to give chart consumers the flexibility to pass the credentials via their own secret, the remote.kubernetes.secret method made the feature as "useful" as possible by allowing the secret to exist anywhere in the cluster, freeing consumers from having to fuss with other methods of getting the secret into the namespace (External Secrets Operator or whatever). This was understandable given the knowledge that the barrier between namespaces had already been busted by the permissions Alloy granted itself. Is that accurate, or is the use of remote.kubernetes.secret (and therefore the need for RBAC) necessary for other reasons?

If having the secrets in the namespace makes it possible to just reference them through environment variables then I would prefer that as the default method. I'm happy to be a sounding board about how to mitigate such a breaking change.