Passing client-side certs through two NGINX servers using MTLS and delivering them to a back-end application

[alnhk]

Is it possible to configure NGINX to pass client-side certificates through two NGINX servers and send the original client-side certificate to destination app?

I've included a diagram below:

Highlights are:

1. curl (or a browser) is configured with client side certs.
2. Mutual authentication is required on both NGINX services.
   3.Trusted certs and requisite CA certs are configured.
3. NGINX takes client side cert and passes it as HTTP header fields.
4. The 2nd NGINX server is actually in a K8S cluster.
5. In the 1st Nginx, if the proxypass to a upstream or service is exhausted or down, we add the mechanism to return HTTP 503.
6. Incase 1st Nginx confirms HTTP 503, the request is passed down to 2nd Nginx (in another region).

FILE: /etc/nginx/nginx.conf (on NGINX-A)
http {
    upstream k8s {
        server nginx-b.k8s.example.com;
    }
    server {
        listen                 443 ssl;
        ssl_verify_client      on;
        ssl_certificate        /etc/nginx/certs/identity.pem;
        ssl_certificate_key    /etc/nginx/certs/identity.key;
        ssl_client_certificate /etc/nginx/certs/ca-bundle.crt;

        location /upstream {
            proxy_pass 'https://nginx-b.k8s.example.com';
            proxy_set_header ssl-client-cert $ssl_client_escaped_cert;
            proxy_set_header ssl-client-subject-dn $ssl_client_s_dn;
            proxy_set_header ssl-client-issuer-dn $ssl_client_i_dn;
            
            proxy_ssl_certificate          /etc/nginx/certs/identity.pem;
            proxy_ssl_certificate_key      /etc/nginx/certs/identity.key;
            proxy_ssl_trusted_certificate  /etc/nginx/certs/ca-bundle.crt;
            proxy_ssl_verify               on;
            proxy_ssl_verify_depth         4;
        }

The ingress object for the application has the following configuration for it.

ingress:
  enabled: true
  annotations:
    # Enable client certificate authentication
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    # Create the secret containing the trusted ca certificates
    nginx.ingress.kubernetes.io/auth-tls-secret: "my-ns/ca-bundle"
    # Specify the verification depth in the client certificates chain
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "4"
    # Specify if certificates are passed to upstream server
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
  tls:
    - secretName: cert-to-use
      hosts:
        - mydns.in.k8s.example.com

looking for solution:

Whenever we hit at Nginx-A, the client certificate does show up, however, when the upstream service under Nginx-A is down or exhausted, spill over/failover to Nginx-B happens, everything is confirmed working, however, at Nginx-B, we are not seeing any client certificate passed down. And prints {\x22error\x22: \x22no client certificate\x22})

Any insight is appreciated w.r.t Nginx-B where we expect to see the client certificate passed down so that mTLS to MTLS execution works properly. The reason for the ask is, if we use "ssl_verify_client" is optional, it works fine all the way from client -> Nginx-A (spill over to Nginx-B), however if we set ssl_verify_client to "on", the spill over to Nginx-B will fail with HTTP 400 No required SSL certificate was sent...

nginx-ingress-version :
nginx version: nginx/1.25.5 (nginx-plus-r32-p1)
based out of the ingress-nginx helm chart 3.6.1
kubernetes version : v1.27.11

ENvironment :
VMWAre
RHEL 8.10
cluster created using kubeadm

[alnhk]

Hello @haywoodsh @brianehlert @vepatel - can you please help to expedite ?