monitoring-flux-infra

This is a work-in-progress FluxCD repository to bring up common monitoring applications and infrastructure for integration across different Kubernetes clusters, depending on the requirements of the respective projects.

Repository structure

.
├── applications
│   ├── alloy
│   ├── kube-prometheus-stack
│   └── loki
├── clusters
│   ├── azure
│   │   └── production
│   └── kind
├── examples
│   ├── azure
│   │   └── production
│   │       ├── base
│   │       │   └── flux-system
│   │       └── monitoring
│   └── kind
│       ├── base
│       │   └── flux-system
│       └── monitoring
├── infrastructure
│   ├── dashboards
│   ├── logs
│   └── monitors
└── scripts

Applications

This directory serves to aggregate all shareable monitoring applications, primarily to implement DRY principles across the various projects at Digital Catapult and reduce the overall replication of YAML throughout the organisation's Git repositories.

For each HelmRelease, there should be two maps: env-values and base-values, corresponding to the environment-specific substitutions for each cluster type and the minimal viable configuration needed for the application.

Within sources.yaml, there are the Helm and OCI repositories for all components needed by the applications.

Clusters

Where there are references here to clusters or environments, they are best thought of as types of deployment. This repository is intended to abstract monitoring components and be a step removed; each project will need to integrate the relevant type of cluster being deployed via a specific kustomization, e.g. "monitoring-sync". This monitoring repository will be an extra GitRepository in any project.

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: monitoring-flux-infra
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  url: https://github.com/digicatapult/monitoring-flux-infra.git

Anticipate including the following monitoring-sqnc.yaml along the base path of new or existing project clusters:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: monitoring-sync
  namespace: monitoring
spec:
  interval: 10m0s
  path: ./applications
  prune: true
  sourceRef:
    kind: GitRepository
    name: monitoring-flux-infra
    namespace: flux-system

A separate env-sync.yaml kustomization should point to the cluster type:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: env-sync
  namespace: monitoring
spec:
  interval: 10m0s
  path: ./clusters/kind
  prune: true
  sourceRef:
    kind: GitRepository
    name: monitoring-flux-infra
    namespace: flux-system

If the project has multiple clusters, then several monitoring-sync.yaml and env-sync.yaml files will be required, residing on the relevant base path.

Examples

Within the examples directory, there are environment-specific demonstrations for different platforms or implementations, e.g. Azure or Kind. These examples have their own flux-system directory, to better illustrate the abstraction. When adapting the examples to a live environment or some other deployment, there should always be a single flux-system GitRepository resource. This repository ought to be used to provide the overlays for the monitoring applications and the accompanying infrastructure configuration for dashboards or additional controllers.

Infrastructure

As with the applications, the complementing infrastructure has been separated so that cluster types only include the dashboards, monitors, and controllers required for that environment. Such controllers might be implemented by Bitnami's cert-manager or nginx-ingress-controller charts. There will likely be occasions where these need to be considered separately from the monitoring stacks proper, hence the option for additional controllers here.

Scripts

There are scripts to create a Kind cluster and install FluxCD to it.

Getting started

Adding new applications

When adding new applications, check that the namespace is available and that no existing releases in the target cluster are using the same name.

A typical application with have the following components:

.
├── kustomization.yaml
├── release.yaml
└── values.yaml

At a minimum, any new files and resources will need to be referenced in the following locations:

• applications/kustomization.yaml, to capture the base values for the new application in the base-values map,
• clusters/**/kustomization.yaml, to allow the target environment to substitute the defaults with values more suited to that cluster type via env-values.

If the application requires additional repositories, then applications/sources.yaml should also be updated with the relevant HelmRepository or OCIRepository.

Adding secrets

Secrets ideally ought to be kept within individual project repositories, as this is intended to be a modular, abstracted monitoring stack to be added to existing clusters as needed.

Managing releases

Semantic versions and other environment-specific configuration can be managed separately when using shared applications, so that updates or reversions are not immediately applied across all projects. Such usage is optional, however. When adding HelmRelease resources, a semantic version could still be specified within the spec there instead.

    spec:
      chart: kube-prometheus-stack
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
      version: ${KUBE_PROMETHEUS_STACK_VERSION}

Above, the environment variable is provided by a kustomization within the target cluster. If all monitoring stacks on all projects can be kept up-to-date, which is likely always true with Kind anyway, then the example below would suffice instead:

    spec:
      chart: kube-prometheus-stack
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
      version: "75.6.0"

Working with shared values

Some attention needs to be given to the precedence of values; the last values file listed for a given HelmRelease will override anything that precedes it. In the majority of use cases, environment-specific values for the cluster should substitute the shared values for a release.

  valuesFrom:
    - kind: ConfigMap
      name: base-values
      valuesKey: kube-prometheus-stack-base-values.yaml
    - kind: ConfigMap
      name: env-values
      valuesKey: kube-prometheus-stack-env-values.yaml

To provide an example of the above, there could be relevant values files in three separate locations:

• applications/kube-prometheus-stack/values.yaml
• clusters/kind/substitutions/kube-prometheus-stack/values.yaml
• clusters/azure/production/substitutions/kube-prometheus-stack/values.yaml

By default, the specific values for each of the above cluster types will always override the shared ones.

Using Renovate for CI

In most cases, Renovate will be used to just update the base layer of applications and the Kind overlays, leaving the Azure overlays untouched. Developers will have the choice of whether to pass staging or production versions through environment variables or ConfigMaps, using .spec.postBuild.Substitute or similar in the corresponding Azure overlay.