Traefik TLS Issues (letsencrypt/cloudflare)

[ianwesleyarmstrong]

Hello!

Thanks for such releasing such a wonderful tool! I've been having some issues with with my Traefik (https more specifically) setup. I'm using Traefik, letsencrypt, and cloudflare with DNS challenges. I'm able to solve the challenges and produce valid certificates (verified in cloudflare/browser), but the problem seems to be coming from the application side? The stock Traefik dashboard works fine, but when accessing several other services (ArgoCD/Grafana/Prometheus), the pages themselves load, but any content within them 404's out. I'm absolutely pulling my hair out on this, so I would really appreciate spot check! I'm happy to offer up any more info if needed. Thanks!

traefik_values

installCRDs: true
globalArguments:
  - "--global.sendanonymoususage=false"
  - "--global.checknewversion=false"
additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--log.level=DEBUG"
deployment:
  kind: DaemonSet
  enabled: true
  annotations: {}
  podAnnotations: {}
  additionalContainers: []
  initContainers: []
ingressRoute:
  dashboard:
    enabled: true
service:
  enabled: true
  type: LoadBalancer
  annotations:
    load-balancer.hetzner.cloud/http-redirect-https: "false"
    load-balancer.hetzner.cloud/name: k3s
    load-balancer.hetzner.cloud/use-private-ip: "true"
    load-balancer.hetzner.cloud/disable-private-ingress: "true"
    load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
    load-balancer.hetzner.cloud/location: "${local.location}"
    load-balancer.hetzner.cloud/type: "${local.load_balancer_type}"
ports:
  web:
    redirectTo: websecure
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
  websecure:
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8

module inputs

    hcloud_token = get_env("HETZNER_API_TOKEN")

    ssh_public_key  = file("~/.ssh/id_ed25519.pub")
    ssh_private_key = file("~/.ssh/id_ed25519")

    network_region = local.region_vars.locals.network_region

    allow_scheduling_on_control_plane = false
    control_plane_nodepools = [
      {
        name        = "control-plane-${local.location}-1"
        server_type = local.control_plane_node_type
        location    = local.location
        labels      = []
        taints      = []
        count       = 1
      },
      {
        name        = "control-plane-${local.location}-2"
        server_type = local.control_plane_node_type
        location    = local.location
        labels      = []
        taints      = []
        count       = 1
      },
      {
        name        = "control-plane-${local.location}-3"
        server_type = local.control_plane_node_type
        location    = local.location
        labels      = []
        taints      = []
        count       = 1
      }
    ]

    control_planes_custom_config = {
      etcd-expose-metrics         = true,
      kube-controller-manager-arg = "bind-address=0.0.0.0",
      kube-proxy-arg              = "metrics-bind-address=0.0.0.0",
      kube-scheduler-arg          = "bind-address=0.0.0.0",
    }

    agent_nodepools = [
      {
        name        = "agent-standard"
        server_type = "cpx21"
        location    = local.location
        labels      = []
        taints      = []
        count       = 0
      }
    ]

    autoscaler_nodepools = [
      {
        name        = "autoscaler-standard"
        server_type = "cpx21"
        location    = local.location
        min_nodes   = 0
        max_nodes   = 5
      },
      {
        name        = "autoscaler-medium"
        server_type = "cpx31"
        location    = local.location
        min_nodes   = 0
        max_nodes   = 4
      },
      {
        name        = "autoscaler-high-cpu"
        server_type = "cpx31"
        location    = local.location
        min_nodes   = 0
        max_nodes   = 3
      },
      {
        name        = "autoscaler-high-mem"
        server_type = "cpx51"
        location    = local.location
        min_nodes   = 0
        max_nodes   = 3
      }
    ]

    ## INGRESS/LB
    base_domain            = "XXXXX.YYY"
    load_balancer_type     = local.load_balancer_type
    load_balancer_location = local.location

    cert_manager_values = <<EOF
installCRDs: true
replicaCount: 3
extraArgs:
  - --dns01-recursive-nameservers=1.1.1.1:53,9.9.9.9:53
  - --dns01-recursive-nameservers-only
podDnsPolicy: None
podDnsConfig:
  nameservers:
    - "1.1.1.1"
    - "9.9.9.9"
  EOF

    traefik_redirect_to_https = false
    traefik_values            = <<EOF
installCRDs: true
globalArguments:
  - "--global.sendanonymoususage=false"
  - "--global.checknewversion=false"
additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--log.level=DEBUG"

deployment:
  kind: DaemonSet
  enabled: true
  annotations: {}
  podAnnotations: {}
  additionalContainers: []
  initContainers: []

ingressRoute:
  dashboard:
    enabled: true

service:
  enabled: true
  type: LoadBalancer
  annotations:
    load-balancer.hetzner.cloud/http-redirect-https: "false"
    load-balancer.hetzner.cloud/name: k3s
    load-balancer.hetzner.cloud/use-private-ip: "true"
    load-balancer.hetzner.cloud/disable-private-ingress: "true"
    load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
    load-balancer.hetzner.cloud/location: "${local.location}"
    load-balancer.hetzner.cloud/type: "${local.load_balancer_type}"
ports:
  web:
    redirectTo: websecure

    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
  websecure:
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
EOF
    create_kubeconfig         = false

ArgoCD ingressRoute

  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`argocd.k3s.XXX.YYY`)
    priority: 10
    services:
    - name: argo-cd-argocd-server
      port: 80
  - kind: Rule
    match: Host(`argocd.k3s.XXX.YYY`) && Headers(`Content-Type`, `application/grpc`)
    priority: 11
    services:
    - name: argo-cd-argocd-server
      port: 80
      scheme: h2c
  tls:
    secretName: argocd-tls

[mysticaltech]

@ianwesleyarmstrong Sorry I missed the discussion, don't hesitate to tag me directly next time. If I were you I would not use the traefik ingress definition format, just the normal kubernetes ingress definition which works equally well with traefik (and you can use traefik middleware too via annotations).

The problem IMHO is in your ingress rules, they do not match any path! See the examples section here for how to do it: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/README.md#examples

Just FYI, for Cloudflare, the way I use it myself is to use their own generated certificate and point the ingress to that, then proxy the SSL request through Cloudflare.

So Cloudflare will generate valid SSL certificates for the world, and connect to your ingress service via its own generated certificate that it only recognizes as valid. It simplifies the whole LetEncrypt step via cert-manager.

[ianwesleyarmstrong]

No worries! Thanks for the pointer about the Path. Ultimately, the thing that fixed it was removing the hardcoded priorities between https/grpc.

  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`argocd.k3s.xxx.yyy`) || Host(`argocd.k3s.xxx.yyy`) && PathPrefix(`/`)
    services:
    - name: argo-cd-argocd-server
      port: 80
  - kind: Rule
    match: (Host(`argocd.k3s.xxx.yyy`) || Host(`argocd.k3s.xxx.yyy`) &&
      PathPrefix(`/`)) && Headers(`Content-Type`, `application/grpc`)
    services:
    - name: argo-cd-argocd-server
      port: 80
      scheme: h2c
  tls:
    secretName: argocd-tls

[mysticaltech]

Perfect, thanks for sharing your experience!