VirtualServer and cert-manager incompatible?

[AlexGoris-KasparSolutions]

Hi,
We recently switched to this ingress controller from the Kubernetes nginx one so we could use Nginx plus and its authentication/authorization features.

After having spent the last few days diving into a lot of documentation, logs, etc, I can only conclude that the VirtualServer(Route) resource types this project defines, are incompatible with the cert-manager project.

To get a secure VirtualServer resource, with a valid TLS certiciate from Let'sEncrypt, I would define something like this:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cafe-tls
spec:
  dnsNames:
    - cafe.example.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: cafe-tls
---
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  tls:
    secret: cafe-tls
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  routes:
  - path: /tea
    action:
      pass: tea

Deploying the certificate will result in cert-manager creating an ingress to solve let'sencrypt's HTTP01 challenges (potentially sensitive values masked out):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  labels:
    acme.cert-manager.io/http-domain: "<masked>"
    acme.cert-manager.io/http-token: "<masked>"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-f2gxv
  namespace: cafe
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: cafe-tls-ztb6n-<masked>-<masked>
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-nrthr
            port:
              number: 8089
        path: /.well-known/acme-challenge/<masked>
        pathType: ImplementationSpecific

The challenge will never be validated however, because as soon as cert-manager creates the ingress, the following can be seen in the logs of the ingress controller:

I0321 12:54:40.550359       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cafe", Name:"cm-acme-http-solver-4pmck", UID:"3c439fec-1ff0-47e6-a6af-f980896aae04", APIVersion:"networking.k8s.io/v1", ResourceVersion:"13526395", FieldPath:""}): type: 'Warning' reason: 'Rejected' All hosts are taken by other resources

When I delete the VirtualServer/cafe resource, and re-create the certificate, all is well, the challenge gets approved and the ingress is deleted again, and I can recreate the VirtualServer/cafe resource and have a working TLS-secured application.

At first I thought, OK, I can just modify our deployment pipeline to first deploy the certificate resource, and deploy the VirtualServer resource after the certificate was successfully validated, but then I thought about certificate renewal. Let'sencrypt issues certs with a 3-month validity, and cert-manager will take care of renewing the certificate shortly beforehand, but if that happens, it will need to create an ingress resource again, and we would be back in the same situation (because the VritualServer resource still exists).

For ingresses, this problem does not exist, because one can add annotations to have cert-manager automatically create the certificate for them, and another annotation to instruct cert-manager to edit the existing ingress in-place instead of creating a new one. This would look like so:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod #Instructs cert-manager to create the certificate resource itself
    acme.cert-manager.io/http01-edit-in-place: "true" #Instructs cert-manager to edit this ingress resource (ingress/cafe-ingress) in-place to validate the letsencrypt challenge, instead of creating a new ingress resource
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: cafe-secret
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        pathType: Prefix
        backend:
          service:
            name: tea-svc
            port:
              number: 80

I doubt I'm the first person running into this, as using VirtualServer resources with cert-manager Certificate resources should be pretty basic thing to do to anyone using this ingress controller, so am I missing something?

I found some issues which were more or less related:

• VirtualServer prevents cert-manager from creating a certificate #2069, especially this comment which seems to confirm cert-manager needs to be aware of the VirtualServer resource.
• Support hard coding the service name for http01 cert-manager/cert-manager#3496 this issue was created before the comment in previous related issue (strangely enough), and seems to request a new feature in order to be able to deal with VirtualServer resources, however the request was rejected

In general, going to this ingress controller from the Kubernetes nginx one, seems like a downgrade, as these issues were not a problem with that one, even without using the acme.cert-manager.io/http01-edit-in-place: "true" annotation, everything worked, it seems that other ingress controller is able to deal with multiple ingresses using the same hostname

[brianehlert]

the cert-manager project works with the ingress resource as expected.
We are actively working on making cert-manager aware of the VirtualServer resource.
This is in development right now.

[Joren-Thijs-KasparSolutions]

@brianehlert thank you for your response. Is there any ticket or issue where we can track the progress? Or any timeline on you can give on this? Like explained above we can make it work for now but in three months everything is going to break :(

[brianehlert]

It will continue to work with the Ingress resource. That won't change.
What will change is the addition of proper support for the VirtualServer resource.

[lucacome]

Initial support for VirtualServer was added in https://github.com/nginxinc/kubernetes-ingress/releases/tag/v2.2.0

[AlexGoris-KasparSolutions]

OK, that sounds nice, I've been looking at that link, and more specifically the related PR (#2572), but can't seem to find how exactly we're supposed to configure a VirtualServer resource so it will get picked up by cert manager. It's also not clear whether I only need to update the ingress controller or whether also a new cert-manager version (perhaps yet to be released?) is required.

Could someone shed some light on this?

[brianehlert]

Release 2.2 makes cert-manager aware of the VirtualServer resource. Simply upgrade to 2.2, the code is in there.
https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualservertlscertmanager

Note: ACME Challenge is a more complex workflow and still in development by our team.

[darkn3rd]

Regarding ACME Challenge? What more is needed? As I understand it, ACME takes care of the complex workflow, and places the fetched certificate in the secret, which VirtualServer resource can use?

[brianehlert]

The cert-manager project added support for the Ingress object itself.
The additional work that we needed to do for our project is add the scaffolding to support our CRDs - which give customers greater control and capabilities beyond Ingress.
This work was completed with the 2.3 release.

To give a bit more detail - the cert-manager project creates an ephemeral Ingress resource to add the ACME challenge path for the period of time it is necessary. This inherently created a configuration clash for customers using our CRDs, as the same hostname would be configured through both an Ingress and VirtualServer - we needed to rectify this clash while maintaining integrity of the configuration.

[Joren-Thijs-KasparSolutions]

@brianehlert I tried implementing this support for virual servers but ran into a problem.

The kuberentes tooling does not seem to recognize these new properties and produces following errors
when running kubectl apply in azure pipelines:

error validating "/home/vsts/work/r1/a/KubernetesManifests/common-virtual-server.yaml": error validating data: ValidationError(VirtualServer.spec.tls): unknown field "cert-manager" in org.nginx.k8s.v1.VirtualServer.spec.tls; if you choose to ignore these errors, turn validation off with --validate=false
error validating "/home/vsts/work/r1/a/KubernetesManifests/keycloak.yaml": error validating data: ValidationError(VirtualServer.spec.tls): unknown field "cert-manager" in org.nginx.k8s.v1.VirtualServer.spec.tls; if you choose to ignore these errors, turn validation off with --validate=false

I first updated our nginx image from 2.1.0 to 2.3.0 like the docs required.

The config used based on the docs and the example:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: keycloak
spec:
  host: keycloak.${HOSTNAME}$
  tls:
    secret: keycloak-tls-secret
    cert-manager:
      cluster-issuer: ${CertManagerClusterIssuerName}$
  upstreams:
    - name: keycloak
      service: keycloak
      port: 8080
      buffer-size: 14k
  routes:
    - path: /auth
      action:
        pass: keycloak

If I try locally and use the --validate=false flag the apply works and the virtual server resource is accepted with a status of valid.
Any ideas on what causes this? it happens both locally and in azure pipelines.

[jasonwilliams14]

@Joren-Thijs-KasparSolutions did you make sure to install/update the latest CRDs for NGINX Ingress controller? That looks to be the case as the field is not recognized and should be resolved with ensuring our latest CRDs are installed into the k8 cluster.

[Joren-Thijs-KasparSolutions]

No updated NGINX but forgot to update the cert-manager CRDs. We have it working now. thanks!

[hubwoop]

Hi I also tried to use cert-manager with a VirtualServer and was not able to get the certificate issued without manually adding the "challenge" service to the VirtualServers upstreams and path to the routes. Cert-manager tried to create a regular k8s Ingress resource which did not work (the challenge could not be completed, and the Ingress resource could not be created properly - I assume because the hostname is already taken by the VirtualServer?). After I added the path of the ingress to my VirtualServer I immediatly got the certificate 🎉. While this is nice, I assume that I should be able to get the certificate without manually editing my VirtualServer right?

This is my VirtualServer without the manual addition of the cert-manager/let's encrypt specific routes/upstreams

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  tls:
    secret: certman-cafe-secret
    cert-manager:
      cluster-issuer: letsencrypt-staging-nginxinc
      common-name: cafe.example.com
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  - name: coffee
    service: coffee-svc
    port: 80
  routes:
  - path: /tea
    action:
      proxy:
        upstream: tea
        rewritePath: /
  - path: /coffee
    action:
      proxy:
        upstream: coffee
        rewritePath: /
  - action:
      return:
        body: |
          Hello World
        code: 200
        type: text/plain
    path: /

[brianehlert]

That is correct. We treat the ephemeral ingress object as a special entity and allow it to drive the path for the challenge response.

Can I ask what version of NGINX Ingress Controller you are running?

I also know that cert-manager just released an update and would like to know that version as well if you don't mind. To make sure we are not out of sync with some change in behavior on their side.

[hubwoop]

Hi thanks for the quick response :)! Yes:

• cert-manager is running v1.8.2
• NGINX Ingress Controller (free not plus):
  • appVersion: 2.4.1
  • chartVersion: 0.15.1

[hubwoop]

One more Info: On my ClusterIssuer.cert-manager.io/v1 named letsencrypt-staging-nginxinc, I have set spec.acme.solvers[0].http01.ingress.class=nginx.org/ingress-controller

[ciarams87]

Hi @hubwoop! I'm unable to reproduce your issue - I have used your VirtualServer spec and the acme challenge is successful. You are correct that the Ingress would have an hostname conflict issue - as Brian mentions above, we treat acme challenge Ingresses as special entities and translate them to a VirtualServerRoute spec and add them to the VirtualServer configuration (so basically we codify the workaround you have figured out 😄).

I believe what may be happening here is that the spec.acme.solvers[0].http01.ingress.class should be set to the IngressClass name rather than the controller. So, if you have an IngressClass deployed with the spec

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: nginx.org/ingress-controller

your ClusterIssuer spec should be set to something like

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-nginxinc
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: my-email.example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: example-issuer-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

I believe the Ingress Controller may not be picking up this challenge because the IngressClass is incorrect.

Hope this helps!

[hubwoop]

Haha perfect 🎉! This definitely helped! And now everything works as expected! I just had to specify the ingressClass name (instead of the controller) on my ClusterIssuer.

Thanks for the help @ciarams87 and @brianehlert ♥!

[darkn3rd]

I don't know if this is useful, but I was able to get cert manager with DNS01 challenge using VirtualServer working. This was with externalDNS as well on GKE w WorkloadIdentity to grant access to CloudDNS: https://faun.pub/extending-gke-with-nginx-ic-2ae86a96de18