Document Let's Encrypt ACME account key secrets for cert-manager

[Copilot]

Users were unclear about how letsencrypt-prod-account-key and letsencrypt-staging-account-key secrets are created. These are auto-generated by cert-manager, not retrieved from Let's Encrypt.

Changes

• New section: "Understanding the ACME Account Key Secret" explaining:

  • cert-manager auto-generates RSA private key on ClusterIssuer creation
  • Secret location in cert-manager namespace
  • Account persistence across restarts

• Backup/restore instructions for disaster recovery:

  # Export
  kubectl get secret letsencrypt-prod-account-key -n cert-manager -o yaml > letsencrypt-account-backup.yaml
  
  # Restore (before creating ClusterIssuer)
  kubectl apply -f letsencrypt-account-backup.yaml

• Troubleshooting for account key issues with common failure modes

• Staging note clarifying staging/prod use separate accounts and secrets

• cert-manager installation instructions: Added step-by-step guide for installing cert-manager, which is required before creating ClusterIssuers. Includes fix for the "no matches for kind ClusterIssuer" error when cert-manager is not installed:

  helm repo add jetstack https://charts.jetstack.io
  helm repo update
  helm install cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --set crds.enabled=true

• Updated prerequisites: Clear installation order documented (cert-manager → ClusterIssuer → overlay)

• Helm ownership conflict troubleshooting: Added documentation for the "invalid ownership metadata" error that occurs when a ClusterIssuer was created manually with kubectl apply and Helm tries to manage it. Explains why ClusterIssuers should be managed separately from application charts (cluster-scoped, shared across releases) and provides solutions including removing local template files or adopting resources into Helm.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[bwalsh]

@copilot
Getting error

"""
[ClusterIssuer](error: resource mapping not found for name: "letsencrypt-prod" namespace: "" from "helm/argo-stack/overlays/ingress-authz-overlay/templates/cluster-issuer-letsencrypt.yaml": no matches for kind "ClusterIssuer" in version "cert-manager.io/v1")
"""

[Copilot]

@copilot
Getting error

...

This error means cert-manager is not installed in your cluster. Install it first:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

# Wait for it to be ready
kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s

Then create the ClusterIssuer, and finally deploy the overlay.

Added installation instructions and troubleshooting to the documentation in ca11026.

[bwalsh]

@copilot
Getting error

"""
helm upgrade --install ingress-authz-overlay
helm/argo-stack/overlays/ingress-authz-overlay
--namespace argo-stack
--set ingressAuthzOverlay.host=${ARGO_HOSTNAME}
Error: UPGRADE FAILED: Unable to continue with update: ClusterIssuer "letsencrypt-prod" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "ingress-authz-overlay"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "argo-stack"
"""

[Copilot]

@copilot
Getting error

...

This error occurs because the ClusterIssuer was created manually with kubectl apply, and now there's a Helm template trying to manage it.

Quick fix: If you have a local cluster-issuer-letsencrypt.yaml file in your templates, remove it:

rm helm/argo-stack/overlays/ingress-authz-overlay/templates/cluster-issuer*.yaml

The chart is designed to reference an existing ClusterIssuer via annotation, not create one. ClusterIssuers should be created separately with kubectl apply before deploying this overlay.

Added troubleshooting section for this error in 25c158c.

[bwalsh]

@copilot

Getting error

"""
1125 02:40:39.668424 1 setup.go:276] "ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" logger="cert-manager.controller" resource_name="letsencrypt-staging" resource_namespace="" resource_kind="ClusterIssuer" resource_version="v1" related_resource_name="letsencrypt-staging-account-key" related_resource_namespace="cert-manager" related_resource_kind="Secret"
I1125 02:40:40.035363 1 setup.go:410] "verified existing registration with ACME server" logger="cert-manager.controller" resource_name="letsencrypt-staging" resource_namespace="" resource_kind="ClusterIssuer" resource_version="v1" related_resource_name="letsencrypt-staging-account-key" related_resource_namespace="cert-manager" related_resource_kind="Secret"
I1125 02:40:42.905362 1 sync.go:446] "certificate resource is not owned by this object. refusing to update non-owned certificate resource for object" logger="cert-manager.controller.ingress-shim" resource_name="ingress-authz-applications" resource_namespace="argo-stack" resource_kind="" resource_version="" related_resource_name="calypr-demo-tls" related_resource_namespace="argo-stack" related_resource_kind="Certificate" related_resource_version="v1"
I1125 02:40:42.909646 1 sync.go:446] "certificate resource is not owned by this object. refusing to update non-owned certificate resource for object" logger="cert-manager.controller.ingress-shim" resource_name="ingress-authz-workflows" resource_namespace="argo-stack" resource_kind="" resource_version="" related_resource_name="calypr-demo-tls" related_resource_namespace="argo-stack" related_resource_kind="Certificate" related_resource_version="v1"
"""