Policy to delete multiple entry on mutating resources

[husnialhamdani]

Summary: Delete multiple entry for a matching name

Hi everyone,

I'm using Kyverno policy to do some automation to an istioOperator for some Kubernetes events / object get triggered,

I have 3 policy here, with the following details:

1. Policy: generate-iop-cr-ingressgateway, be triggered for every new namespace that start with product-* on the name.
2. Policy: istio-mesh-manager-add, Add 2 diffrent type of entry at a time (opentelemetry-{{ request.object.metadata.name }} and oauth2-proxy-{{ request.object.metadata.name }}).
3. Policy: remove-auth-provider, remove the 2 entry created on policy number 2, with current policy only able to delete 1.

Steps to reproduce:

Create these prerequisites object before triggering the policy:

1. The IstioOpertor object to be mutated.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: control-plane
  namespace: istio-system
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        zipkin:
          address: zipkin.monitoring.svc.cluster.local:9411
  values:
    global:
      proxy:
        autoInject: enabled
        resources:
          requests:
            cpu: 20m
            memory: 128Mi
      useMCP: false
      jwtPolicy: first-party-jwt
  addonComponents:
    pilot:
      enabled: true
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
    pilot:
      k8s:
        resources:
          requests:
            memory: 2Gi
        hpaSpec:
          maxReplicas: 2

2. The aggregated role required to mutate IstioOperator.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/name: kyverno
  name: kyverno:all-istiooperators
rules:
- apiGroups:
  - install.istio.io
  resources:
  - istiooperators
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

3. Apply the generate-iop-cr-ingressgateway policy.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-iop-cr-ingressgateway
spec:
  generateExistingOnPolicyUpdate: true
  rules:
  - name: generate-iop-cr-ingressgateway
    match:
      any:
      - resources:
          kinds:
          - Namespace
          names:
          - "product-*"
          selector:
            matchLabels:
              kyverno-watch: "true"
    generate:
      synchronize: true
      apiVersion: install.istio.io/v1alpha1
      kind: IstioOperator
      name: "{{ request.object.metadata.name }}"
      namespace: istio-system
      data:
        metadata:
          ownerReferences:
          - apiVersion: v1
            kind: Namespace
            name: "{{ request.object.metadata.name }}"
            uid: "{{ request.object.metadata.uid }}"
        spec:
          components:
            base:
              enabled: false
            cni:
              enabled: false
            egressGateways:
            - enabled: false
              name: istio-egressgateway
            ingressGateways:
            - enabled: false
              name: istio-ingressgateway
            - enabled: true
              name: "{{ request.object.metadata.name }}"
              label:
                istio: "{{ request.object.metadata.name }}"
                multitenant: enabled
            istiodRemote:
              enabled: false
            pilot:
              enabled: false
          hub: gcr.io/istio-release
          profile: default
          tag: 1.12.9
          values:
            gateways:
              istio-ingressgateway:
                autoscaleEnabled: true
                env: {}
                name: "{{ request.object.metadata.name }}"
            global:
              proxy:
                autoInject: enabled
              useMCP: false
              jwtPolicy: first-party-jwt

5. Apply the istio-mesh-manager-add policy.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: "istio-mesh-manager-add"
spec:
  background: false
  mutateExistingOnPolicyUpdate: false
  rules:
    - name: "add-provider"
      match:
        any:
        - resources:
            kinds:
            - IstioOperator
            names:
            - "product-*"
            namespaces:
            - istio-system
      preconditions:
        all:
        - key: "{{ request.operation }}"
          operator: Equals
          value: CREATE
      mutate:
        targets:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: control-plane
          namespace: istio-system
        patchesJson6902: |-
          - path: "/spec/meshConfig/extensionProviders/-"
            op: add
            value: {
                "name": "oauth2-proxy-{{ request.object.metadata.name }}",
                "envoyExtAuthzHttp": {
                    "service": "oauth2-proxy.{{ request.object.metadata.name }}.svc.cluster.local",
                    "port": "80",
                    "includeHeadersInCheck": [
                        "authorization",
                        "cookie"
                    ],
                    "headersToUpstreamOnAllow": [
                        "x-forwarded-access-token",
                        "authorization",
                        "path",
                        "x-auth-request-user",
                        "x-auth-request-email",
                        "x-auth-request-access-token"
                    ],
                    "headersToDownstreamOnDeny": [
                        "content-type",
                        "set-cookie"
                    ]
                }
            }
          - path: "/spec/meshConfig/extensionProviders/-"
            op: add
            value: {
              "name": "opentelemetry-{{ request.object.metadata.name }}",
              "opencensus": {
                  "context": [
                      "B3"
                  ],
                  "port": 55678,
                  "service": "opentelemetry-{{ request.object.metadata.name }}-collector.istio-system.svc.cluster.local"
              }
            }

6. Apply the istio-mesh-manager-delete policy.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: "istio-mesh-manager-delete"
spec:
  background: true
  mutateExistingOnPolicyUpdate: false
  rules:
    - name: "remove-auth-provider"
      match:
        any:
        - resources:
            kinds:
            - IstioOperator
            names:
            - "product-*"
            namespaces:
            - istio-system
      preconditions:
        all:
        - key: "{{ request.operation }}"
          operator: Equals
          value: DELETE
      mutate:
        targets:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: control-plane
          namespace: istio-system
        foreach: 
        - list: target.spec.meshConfig.extensionProviders[]
          preconditions:
            any:
            - key: "{{`{{ element.name }}`}}"
              operator: Equals
              value: "oauth2-proxy-{{`{{ request.object.metadata.name }}`}}"
            - key: "{{`{{ element.name }}`}}"
              operator: Equals
              value: "opentelemetry-{{`{{ request.object.metadata.name }}`}}"
          patchesJson6902: |-
            - path: /spec/meshConfig/extensionProviders/{{ elementIndex }}
              op: remove

7. Trigger the generate-iop-cr-ingressgateway and istio-mesh-manager-add policy with:

kubectl create ns product-a
kubectl label namespaces product-a kyverno-watch=true

by creating that namespace and label, the 2 Kyverno policy (generate-iop-cr-ingressgateway and istio-mesh-manager-add) will generate an IstioOperator object and add the 2 different type of entry at a time (the oauth2-proxy-{{ request.object.metadata.name }} and opentelemetry-{{ request.object.metadata.name }})

8. Trigger the delete operation (istio-mesh-manager-delete)

kubectl delete ns product-a

The current delete policy only remove 1 of the matching entry (from total 2)

Looking for an advise to delete 2 entry at a time when there is a delete trigger to the namespace name.

[chipzoller]

You have both of these configured as "mutate existing" rules, meaning they will take effect at the time you create the policy, on EXISTING resources. From your description, it's not clear if this is what you want. You should only use this rule type if you do not want to mutate based upon admission requests. What is your desire here?

[husnialhamdani]

Is it possible to delete the 2 entry in single rule?

[chipzoller]

You should use two loops.

[husnialhamdani]

With this sample entry

extensionProviders:
    - envoyExtAuthzHttp:
        headersToDownstreamOnDeny:
        - content-type
        - set-cookie
      name: oauth2-proxy-product-a
    - name: opentelemetry-product-a
      opencensus:
        context:
        - B3
        port: 55678
        service: opentelemetry-product-a-collector.monitoring.svc.cluster.local
    - envoyExtAuthzHttp:
        headersToDownstreamOnDeny:
        - content-type
        - set-cookie
      name: oauth2-proxy-product-b
    - name: opentelemetry-product-b
      opencensus:
        context:
        - B3
        port: 55678
        service: opentelemetry-product-b-collector.monitoring.svc.cluster.local

How to modify the below rules to use two loops?

foreach: 
        - list: "target.spec.meshConfig.extensionProviders[]"
          preconditions:
            any:
            - key: "{{`{{ element.name }}`}}"
              operator: Equals
              value: "oauth2-proxy-{{`{{ request.object.metadata.name }}`}}"
            - key: "{{`{{ element.name }}`}}"
              operator: Equals
              value: "opentelemetry-{{`{{ request.object.metadata.name }}`}}"
          patchesJson6902: |-
            - path: /spec/meshConfig/extensionProviders/{{`{{ elementIndex }}`}}
              op: remove

[husnialhamdani]

As of now, the temporary solution is like this:

        foreach: 
        - list: "target.spec.meshConfig.extensionProviders[]"
          preconditions:
            any:
            - key: "{{`{{ element.name }}`}}"
              operator: Equals
              value: "oauth2-proxy-{{`{{ request.object.metadata.name }}`}}"
          patchesJson6902: |-
            - path: /spec/meshConfig/extensionProviders/{{`{{ elementIndex }}`}}
              op: remove
            - path: /spec/meshConfig/extensionProviders/{{`{{ elementIndex }}`}}
              op: remove

since the opentelemetry-{{ request.object.metadata.name }} entry is always next to the oauth2-proxy-{{ request.object.metadata.name }}, I put the remove patch twice

but still curious with the two loops idea, can you share @chipzoller?

[chipzoller]

You may not understand how foreach works. The expression you put in the list field will produce an array of extensionProviders and Kyverno will iterate over each one separately. So if there are 5 extensionProviders it will iterate 5 times with each iteration only having a single element. Every one has a name field, and the name can only be one option. So you don't actually need multiple loops, you just use preconditions as you are doing to identify which extensionProviders entry you want to remove. Preconditions under an any[] block are an array, so this is where you list a second precondition that matches on the other name.

[chipzoller]

I was able to reproduce this bug on 1.10.0-alpha.2 and have logged #7051 and scheduled for 1.10.

[husnialhamdani]

Thanks for letting me know if there's a bug on that

[husnialhamdani]

With current add policy, the entry is duplicate, what could be the issue for that?

extensionProviders:
   - envoyExtAuthzHttp:
       headersToDownstreamOnDeny:
       - content-type
       - set-cookie
       headersToUpstreamOnAllow:
       - x-forwarded-access-token
       - authorization
       - path
       - x-auth-request-user
       - x-auth-request-email
       - x-auth-request-access-token
       includeHeadersInCheck:
       - authorization
       - cookie
       port: "80"
       service: oauth2-proxy.product-a.svc.cluster.local
     name: oauth2-proxy-product-a
   - name: opentelemetry-product-a
     opencensus:
       context:
       - B3
       port: 55678
       service: opentelemetry-product-a-collector.istio-system.svc.cluster.local
   - envoyExtAuthzHttp:
       headersToDownstreamOnDeny:
       - content-type
       - set-cookie
       headersToUpstreamOnAllow:
       - x-forwarded-access-token
       - authorization
       - path
       - x-auth-request-user
       - x-auth-request-email
       - x-auth-request-access-token
       includeHeadersInCheck:
       - authorization
       - cookie
       port: "80"
       service: oauth2-proxy.product-a.svc.cluster.local
     name: oauth2-proxy-product-a
   - name: opentelemetry-product-a
     opencensus:
       context:
       - B3
       port: 55678
       service: opentelemetry-product-a-collector.istio-system.svc.cluster.local

[chipzoller]

Only reason I can see is there are multiple CREATE requests for the same trigger.

[husnialhamdani]

is it possible to add a conditional check to ignore if the entry already exist?

[husnialhamdani]

or just add precondition there to check if it already exist or not?

[husnialhamdani]

It's strange, I downgraded the Kyverno to 1.9.0 (from 1.9.2) and no longer create a duplicate entry

[chipzoller]

Check on the 1.10.0-alpha.2 code that's now available. You can install with a Helm chart.

[chipzoller]

Is this question resolved?