Grafana Operator: How to manage dashboards across multiple organizations in the same instance

[Guerlielton]

Hello everyone,

I’m using the Grafana Operator (v5.19.4) to manage dashboards in a Kubernetes environment, and I have a question about handling multiple organizations.

My current setup:

• A single Grafana instance (kube-prometheus-stack)
• Multiple organizations within that instance: main, dev
• Dashboards managed via ConfigMaps with GrafanaDashboard CRs
• Deployments via ArgoCD
• Grafana Operator configured as an external instance:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: kube-prometheus-stack-grafana
  namespace: monitoring
spec:
  external:
    url: http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80
    adminUser:
      name: kube-prometheus-stack-grafana
      key: admin-user
    adminPassword:
      name: kube-prometheus-stack-grafana
      key: admin-password

Here’s an example of one of my GrafanaDashboard CRs:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: hubble-network-overview-namespace
  namespace: default
  labels:
    app: kube-prometheus-stack-grafana
spec:
  allowCrossNamespaceImport: true
  configMapRef:
    name: hubble-network-overview-namespace
    key: hubble-network-overview-namespace.json
  folder: "Cilium"
  instanceSelector:
    matchLabels:
      app: grafana
  resyncPeriod: 10m

Main question:
Is it possible to configure the Grafana Operator to automatically replicate GrafanaDashboard CRs across the different organizations (main, dev) within the same instance?

Observed behavior (possible bug?):

1. At ArgoCD deploy time: If I’m logged into organization main when the CR is applied, the dashboard is created there.
2. During the resyncPeriod (10m): If I switch to dev and wait for resync, the dashboard appears in dev as well.
3. It seems the operator applies the dashboard in whichever organization the admin user is currently logged into at sync time.

I couldn’t find any documentation on this— is this normal? Does the operator truly depend on the active session’s organization context? Why does it replicate as I navigate between orgs?

What I’ve already tried/researched:

• Checked X-Grafana-Org-Id header via spec.client.headers (but it targets only one org)
• Found issue Support for orgId in GrafanaDashboard #525 requesting orgId support for GrafanaDashboard CRs (closed)
• Read recommendation to use multiple instances instead of orgs
• Tested resync behavior across orgs and observed replication
• Observed behavior during ArgoCD deployments

[Baarsgaard]

As of right now, there's no intention to actively support multiple orgs.
That does however not mean it's impossible.
The operator treats each Grafana CR as a unique instance, so if you're reusing credentials, you'll end up in your current scenario where it follows the admin user.

If you want to use basic authentication, you can "lock" the Operator to a specific org by setting X-Grafana-Org-Id, then regardless of the org you logged into, it will always target the id in the header.

The way around this is by creating a Grafana CR per org 🙂

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: main-kube-prometheus-stack-grafana
  namespace: monitoring
  labels:
    app: kube-prometheus-stack-grafana
spec:
  client:
    headers:
      X-Grafana-Org-Id: 1
  external:
    url: http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80
    adminUser:
      name: kube-prometheus-stack-grafana
      key: admin-user
    adminPassword:
      name: kube-prometheus-stack-grafana
      key: admin-password
---
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: dev-kube-prometheus-stack-grafana
  namespace: default # Namespaces can be different as long as the URL is valid and the operator can reach it.
  labels:
    app: kube-prometheus-stack-grafana
spec:
  client:
    headers:
      X-Grafana-Org-Id: 2
  external:
    url: http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80
    adminUser:
      name: kube-prometheus-stack-grafana
      key: admin-user
    adminPassword:
      name: kube-prometheus-stack-grafana
      key: admin-password

Then the GrafanaDashboard should match both via:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: hubble-network-overview-namespace
  namespace: default
spec:
  allowCrossNamespaceImport: true
  configMapRef:
    name: hubble-network-overview-namespace
    key: hubble-network-overview-namespace.json
  folder: "Cilium"
  instanceSelector:
    matchLabels:
      app: kube-prometheus-stack-grafana

Did not test the following:
IIRC Grafana service accounts only exists within an org.
You can achieve a similar effect by creating a service account in each org and using APIKey auth as well.
Then the identity determines the org, not the header, but it still requires multiple Grafana CRs

[Guerlielton]

Thanks for that @Baarsgaard

[Baarsgaard]

I forgot to mention, using .spec.folder: <name> will result in the folder receiving a unique ID for each org if created by the GrafanaDashboard.
Creating a GrafanaFolder and referencing it with either .spec.folderUID or .spec.folderRef should stabilize it.
.spec.folderRef only works within the same namespace as the GrafanaDashboard