Document Let's Encrypt ACME account key secrets for cert-manager

helm/argo-stack/overlays/ingress-authz-overlay/README.md

@@ -62,4 +62,23 @@ See the [User Guide](docs/authz-ingress-user-guide.md) for architecture diagrams
 - Kubernetes 1.19+
 - Helm 3.x
 - NGINX Ingress Controller
-- cert-manager (for TLS)
+- cert-manager (for TLS) - **must be installed before deploying this overlay**
+
+### Installing cert-manager
+
+If you see `no matches for kind "ClusterIssuer"`, cert-manager is not installed:
+
+```bash
+# Install cert-manager
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+helm install cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --set crds.enabled=true
+
+# Wait for cert-manager to be ready
+kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s
+```
+
+See the [User Guide](docs/authz-ingress-user-guide.md) for complete setup instructions including ClusterIssuer configuration.

helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md

@@ -55,6 +55,36 @@ sequenceDiagram
 
 This overlay uses [cert-manager](https://cert-manager.io/) to automatically provision and renew TLS certificates from [Let's Encrypt](https://letsencrypt.org/).
 
+### Installing cert-manager
+
+**cert-manager must be installed before creating ClusterIssuers or deploying this overlay.**
+
+If you see an error like:
+```
+no matches for kind "ClusterIssuer" in version "cert-manager.io/v1"
+```
+This means cert-manager is not installed. Install it first:
+
+```bash
+# Add the Jetstack Helm repository
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+
+# Install cert-manager with CRDs
+helm install cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --set crds.enabled=true
+
+# Verify cert-manager is running
+kubectl get pods -n cert-manager
+```
+
+Wait for all cert-manager pods to be `Running` before proceeding:
+```bash
+kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s
+```
+
 ### How It Works
 
 ```mermaid
@@ -111,6 +141,58 @@ spec:
 kubectl apply -f cluster-issuer.yaml
 ```
 
+### Understanding the ACME Account Key Secret
+
+The `privateKeySecretRef` (e.g., `letsencrypt-prod-account-key` or `letsencrypt-staging-account-key`) specifies where cert-manager stores the ACME account private key. **You do NOT need to create this secret manually** — cert-manager handles it automatically.
+
+#### How It Works
+
+1. **First-Time Setup**: When you create the ClusterIssuer, cert-manager:
+   - Generates a new RSA private key
+   - Registers a new account with Let's Encrypt using your email
+   - Stores the private key in the specified secret (in the `cert-manager` namespace)
+
+2. **Secret Location**: The secret is created in the same namespace as cert-manager (typically `cert-manager`):
+   ```bash
+   # View the account key secret
+   kubectl get secret letsencrypt-prod-account-key -n cert-manager
+
+   # Describe to see metadata
+   kubectl describe secret letsencrypt-prod-account-key -n cert-manager
+   ```
+
+3. **Account Persistence**: The account key persists across cert-manager restarts. As long as the secret exists, cert-manager will reuse the same Let's Encrypt account.
+
+#### Backing Up the ACME Account Key
+
+For disaster recovery, you may want to back up the account key:
+
+```bash
+# Export the account key secret
+kubectl get secret letsencrypt-prod-account-key -n cert-manager -o yaml > letsencrypt-account-backup.yaml
+
+# To restore in a new cluster (before creating ClusterIssuer)
+kubectl apply -f letsencrypt-account-backup.yaml
+```
+
+> **Note**: Keep the backup secure — this key provides access to your Let's Encrypt account and all its certificates.
+
+#### Troubleshooting Account Key Issues
+
+If the account key secret is not being created:
+
+```bash
+# Check cert-manager controller logs
+kubectl logs -n cert-manager -l app.kubernetes.io/component=controller
+
+# Check ClusterIssuer status
+kubectl describe clusterissuer letsencrypt-prod
+```
+
+Common issues:
+- **ACME Registration Failed**: Check your email address is valid and you can reach Let's Encrypt's API
+- **Secret Not Found in Expected Namespace**: The secret is created in the cert-manager namespace, not your application namespace
+
 ### Configuration Options
 
 | Setting | Description | Default |
@@ -140,6 +222,8 @@ spec:
             class: nginx
 ```
 
+> **Note**: The `letsencrypt-staging-account-key` secret is also auto-generated by cert-manager, just like the production key. Staging and production use separate accounts and secrets.
+
 Then configure the overlay to use it:
 
 ```yaml
@@ -182,17 +266,65 @@ kubectl get challenges -A
 ```
 
 Common issues:
+- **cert-manager not installed**: If you see `no matches for kind "ClusterIssuer"`, install cert-manager first (see [Installing cert-manager](#installing-cert-manager))
+- **Helm ownership conflict**: If you see `invalid ownership metadata; label validation error`, the ClusterIssuer was created outside of Helm. See [Helm Ownership Conflict](#helm-ownership-conflict) below.
 - **Domain not reachable**: Ensure your domain's DNS points to the ingress controller's external IP
 - **Rate limited**: Use `letsencrypt-staging` for testing to avoid production rate limits
 - **Challenge failed**: Check that port 80 is accessible for HTTP-01 challenges
 
+### Helm Ownership Conflict
+
+If you get an error like:
+```
+Error: UPGRADE FAILED: Unable to continue with update: ClusterIssuer "letsencrypt-prod" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by"
+```
+
+This happens when:
+1. The ClusterIssuer was created manually with `kubectl apply`
+2. A Helm chart template tries to create/manage the same ClusterIssuer
+
+**Solution**: ClusterIssuers should be managed **outside** of this Helm chart:
+
+```bash
+# Option 1: Keep the manually created ClusterIssuer (recommended)
+# Simply don't include cluster-issuer templates in the chart
+# This overlay already follows this pattern - it references the ClusterIssuer
+# via annotation but doesn't create it
+
+# Option 2: If you have a local cluster-issuer template file, remove it
+rm helm/argo-stack/overlays/ingress-authz-overlay/templates/cluster-issuer*.yaml
+
+# Option 3: To adopt an existing resource into Helm (advanced)
+# Add Helm labels and annotations to the existing ClusterIssuer:
+kubectl annotate clusterissuer letsencrypt-prod \
+  meta.helm.sh/release-name=ingress-authz-overlay \
+  meta.helm.sh/release-namespace=argo-stack
+kubectl label clusterissuer letsencrypt-prod \
+  app.kubernetes.io/managed-by=Helm
+```
+
+**Why ClusterIssuers are managed separately**: ClusterIssuers are cluster-scoped resources that affect the entire cluster, not just one namespace. Including them in application-specific Helm charts causes conflicts when:
+- Multiple applications need the same ClusterIssuer
+- The ClusterIssuer already exists (created by a previous deployment or another chart)
+- Different teams deploy applications that reference the same issuer
+
+This chart references the ClusterIssuer via annotation (`cert-manager.io/cluster-issuer`) but leaves its lifecycle management to cluster administrators.
+
 ## Installation
 
 ### Prerequisites
 
-- Kubernetes cluster with NGINX Ingress Controller
-- cert-manager installed and configured with a ClusterIssuer (e.g., `letsencrypt-prod`)
-- Helm 3.x
+Before installing this overlay, ensure you have:
+
+1. **Kubernetes cluster** (1.19+) with NGINX Ingress Controller installed
+2. **cert-manager installed** (see [Installing cert-manager](#installing-cert-manager) above)
+3. **ClusterIssuer created** (see [ClusterIssuer: letsencrypt-prod](#clusterissuer-letsencrypt-prod))
+4. **Helm 3.x** installed locally
+
+**Installation Order**:
+1. Install cert-manager
+2. Create ClusterIssuer
+3. Install this overlay
 
 ### Install the Overlay