Is it Observable

Episode : Admission Controller

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to Admission Controller

This tutorial will use the admission Controller and collect observability data related to the policy generated.

We will also utilize the following components :

• The OpenTelemetry Operator
• Dynatrace Operator to report the health of the cluster
• OpenTelemetry Demo
• Unguard

All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• curl
• Helm
• Clustter with the Admission controller enabled

Getting started

1. Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

3. Spinup a k8S cluster with the Admission Controller enabled

NAME=isobs-admissioncontroller

  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          enable-admission-plugins: "MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ValidatingAdmissionPolicy"
          feature-gates: "MutatingAdmissionPolicy=true,DeclarativeValidation=true,DeclarativeValidationTakeover=true"
          runtime-config: "admissionregistration.k8s.io/v1alpha1=true"
          audit-policy-file: /etc/kubernetes/audit-policy.yaml
          audit-log-path: /var/log/kubernetes/audit.log
          audit-log-maxage: "30"
          audit-log-maxbackup: "10"
          audit-log-maxsize: "100"
        extraVolumes:
          - name: audit-policy
            hostPath: /etc/kubernetes/audit-policy.yaml
            mountPath: /etc/kubernetes/audit-policy.yaml
            readOnly: true

          - name: tracing-config
            hostPath: /etc/kubernetes/tracing
            mountPath: /etc/kubernetes/tracing
            readOnly: true
            pathType: DirectoryOrCreate


          - name: audit-logs
            hostPath: /var/log/kubernetes
            mountPath: /var/log/kubernetes
            pathType: DirectoryOrCreate
      files:
      - path: /etc/kubernetes/audit-policy.yaml
        content: |
          apiVersion: audit.k8s.io/v1
          kind: Policy
          rules:
            # Log all admission controller decisions at the RequestResponse level
            - level: RequestResponse
              stages: ["ResponseComplete"]
              resources:
              - group: "admissionregistration.k8s.io"
                resources: ["validatingadmissionpolicies", "mutatingadmissionpolicies"]

            # Log all changes made by mutating policies
            - level: RequestResponse
              verbs: ["create", "update", "patch"]
              resources:
              - group: ""
                resources: ["pods", "deployments", "services", "configmaps", "secrets"]

            # Capture annotations added by admission policies
            - level: RequestResponse
              verbs: ["create", "update"]
              resources:
              - group: ""
                resources: ["pods"]
              omitStages:
              - "RequestReceived"

            # Default fallback rule
            - level: Metadata

      - path: /etc/kubernetes/tracing/apiserver-tracing.yaml
        content: |
          apiVersion: apiserver.config.k8s.io/v1beta1
          kind: TracingConfiguration
          samplingRatePerMillion: 100
          endpoint: "otel-collector.default.svc.cluster.local:4317"

      - path: /etc/kubernetes/tracing/kubelet-tracing.yaml
        content: |
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletTracingConfiguration
          samplingRatePerMillion: 100
          endpoint: "otel-collector.default.svc.cluster.local:4317"

Deploy most of the components

The application will deploy the entire environment:

chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" 

Dashboard

Let's deploy the dashboard located : dynatrace/Kyverno.json

In dynatrace , Open The Dashboard application and click on upload

This dashboard will keep track the usage te APIServer and the policy deployed using:

• metrics from the apiserver
• Audit logs