From 34592ed5bde27cc21d0e7e8b98ca01f5b3d8fdea Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 25 Nov 2025 01:08:41 +0000
Subject: [PATCH 1/6] Initial plan


From 67160df4cbdee07ed7cac865a9835547a7935d9d Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 25 Nov 2025 01:11:23 +0000
Subject: [PATCH 2/6] Add documentation for Let's Encrypt ACME account key
 secrets

Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
---
 .../docs/authz-ingress-user-guide.md          | 54 +++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
index a54d5aed..c18485c8 100644
--- a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
+++ b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
@@ -111,6 +111,58 @@ spec:
 kubectl apply -f cluster-issuer.yaml
 ```
 
+### Understanding the ACME Account Key Secret
+
+The `privateKeySecretRef` (e.g., `letsencrypt-prod-account-key` or `letsencrypt-staging-account-key`) specifies where cert-manager stores the ACME account private key. **You do NOT need to create this secret manually** — cert-manager handles it automatically.
+
+#### How It Works
+
+1. **First-Time Setup**: When you create the ClusterIssuer, cert-manager:
+   - Generates a new RSA private key
+   - Registers a new account with Let's Encrypt using your email
+   - Stores the private key in the specified secret (in the `cert-manager` namespace)
+
+2. **Secret Location**: The secret is created in the same namespace as cert-manager (typically `cert-manager`):
+   ```bash
+   # View the account key secret
+   kubectl get secret letsencrypt-prod-account-key -n cert-manager
+   
+   # Describe to see metadata
+   kubectl describe secret letsencrypt-prod-account-key -n cert-manager
+   ```
+
+3. **Account Persistence**: The account key persists across cert-manager restarts. As long as the secret exists, cert-manager will reuse the same Let's Encrypt account.
+
+#### Backing Up the ACME Account Key
+
+For disaster recovery, you may want to back up the account key:
+
+```bash
+# Export the account key secret
+kubectl get secret letsencrypt-prod-account-key -n cert-manager -o yaml > letsencrypt-account-backup.yaml
+
+# To restore in a new cluster (before creating ClusterIssuer)
+kubectl apply -f letsencrypt-account-backup.yaml
+```
+
+> **Note**: Keep the backup secure — this key provides access to your Let's Encrypt account and all its certificates.
+
+#### Troubleshooting Account Key Issues
+
+If the account key secret is not being created:
+
+```bash
+# Check cert-manager controller logs
+kubectl logs -n cert-manager -l app.kubernetes.io/component=controller
+
+# Check ClusterIssuer status
+kubectl describe clusterissuer letsencrypt-prod
+```
+
+Common issues:
+- **ACME Registration Failed**: Check your email address is valid and you can reach Let's Encrypt's API
+- **Secret Not Found in Expected Namespace**: The secret is created in the cert-manager namespace, not your application namespace
+
 ### Configuration Options
 
 | Setting | Description | Default |
@@ -140,6 +192,8 @@ spec:
             class: nginx
 ```
 
+> **Note**: The `letsencrypt-staging-account-key` secret is also auto-generated by cert-manager, just like the production key. Staging and production use separate accounts and secrets.
+
 Then configure the overlay to use it:
 
 ```yaml

From ca11026ae5026bc0839668f689ee5515a302c383 Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 25 Nov 2025 01:24:07 +0000
Subject: [PATCH 3/6] Add cert-manager installation instructions and
 troubleshooting

Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
---
 .../overlays/ingress-authz-overlay/README.md  | 21 ++++++++-
 .../docs/authz-ingress-user-guide.md          | 45 +++++++++++++++++--
 2 files changed, 62 insertions(+), 4 deletions(-)

diff --git a/helm/argo-stack/overlays/ingress-authz-overlay/README.md b/helm/argo-stack/overlays/ingress-authz-overlay/README.md
index 02a89464..d07fe2c1 100644
--- a/helm/argo-stack/overlays/ingress-authz-overlay/README.md
+++ b/helm/argo-stack/overlays/ingress-authz-overlay/README.md
@@ -62,4 +62,23 @@ See the [User Guide](docs/authz-ingress-user-guide.md) for architecture diagrams
 - Kubernetes 1.19+
 - Helm 3.x
 - NGINX Ingress Controller
-- cert-manager (for TLS)
+- cert-manager (for TLS) - **must be installed before deploying this overlay**
+
+### Installing cert-manager
+
+If you see `no matches for kind "ClusterIssuer"`, cert-manager is not installed:
+
+```bash
+# Install cert-manager
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+helm install cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --set crds.enabled=true
+
+# Wait for cert-manager to be ready
+kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s
+```
+
+See the [User Guide](docs/authz-ingress-user-guide.md) for complete setup instructions including ClusterIssuer configuration.
diff --git a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
index c18485c8..a8652eea 100644
--- a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
+++ b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
@@ -55,6 +55,36 @@ sequenceDiagram
 
 This overlay uses [cert-manager](https://cert-manager.io/) to automatically provision and renew TLS certificates from [Let's Encrypt](https://letsencrypt.org/).
 
+### Installing cert-manager
+
+**cert-manager must be installed before creating ClusterIssuers or deploying this overlay.**
+
+If you see an error like:
+```
+no matches for kind "ClusterIssuer" in version "cert-manager.io/v1"
+```
+This means cert-manager is not installed. Install it first:
+
+```bash
+# Add the Jetstack Helm repository
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+
+# Install cert-manager with CRDs
+helm install cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --set crds.enabled=true
+
+# Verify cert-manager is running
+kubectl get pods -n cert-manager
+```
+
+Wait for all cert-manager pods to be `Running` before proceeding:
+```bash
+kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s
+```
+
 ### How It Works
 
 ```mermaid
@@ -236,6 +266,7 @@ kubectl get challenges -A
 ```
 
 Common issues:
+- **cert-manager not installed**: If you see `no matches for kind "ClusterIssuer"`, install cert-manager first (see [Installing cert-manager](#installing-cert-manager))
 - **Domain not reachable**: Ensure your domain's DNS points to the ingress controller's external IP
 - **Rate limited**: Use `letsencrypt-staging` for testing to avoid production rate limits
 - **Challenge failed**: Check that port 80 is accessible for HTTP-01 challenges
@@ -244,9 +275,17 @@ Common issues:
 
 ### Prerequisites
 
-- Kubernetes cluster with NGINX Ingress Controller
-- cert-manager installed and configured with a ClusterIssuer (e.g., `letsencrypt-prod`)
-- Helm 3.x
+Before installing this overlay, ensure you have:
+
+1. **Kubernetes cluster** (1.19+) with NGINX Ingress Controller installed
+2. **cert-manager installed** (see [Installing cert-manager](#installing-cert-manager) above)
+3. **ClusterIssuer created** (see [ClusterIssuer: letsencrypt-prod](#clusterissuer-letsencrypt-prod))
+4. **Helm 3.x** installed locally
+
+**Installation Order**:
+```
+1. Install cert-manager → 2. Create ClusterIssuer → 3. Install this overlay
+```
 
 ### Install the Overlay
 

From 669b3f110dffbfd538022eaa2f684e5387113c64 Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 25 Nov 2025 01:24:56 +0000
Subject: [PATCH 4/6] Fix installation order list formatting

Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
---
 .../ingress-authz-overlay/docs/authz-ingress-user-guide.md  | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
index a8652eea..80993efb 100644
--- a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
+++ b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
@@ -283,9 +283,9 @@ Before installing this overlay, ensure you have:
 4. **Helm 3.x** installed locally
 
 **Installation Order**:
-```
-1. Install cert-manager → 2. Create ClusterIssuer → 3. Install this overlay
-```
+1. Install cert-manager
+2. Create ClusterIssuer
+3. Install this overlay
 
 ### Install the Overlay
 

From 25c158c58d452cdc0036e362470aa87c3b9b7aaf Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 25 Nov 2025 01:35:05 +0000
Subject: [PATCH 5/6] Add troubleshooting for Helm ownership conflict with
 ClusterIssuer

Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
---
 .../docs/authz-ingress-user-guide.md          | 39 +++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
index 80993efb..3470d6ea 100644
--- a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
+++ b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
@@ -267,10 +267,49 @@ kubectl get challenges -A
 
 Common issues:
 - **cert-manager not installed**: If you see `no matches for kind "ClusterIssuer"`, install cert-manager first (see [Installing cert-manager](#installing-cert-manager))
+- **Helm ownership conflict**: If you see `invalid ownership metadata; label validation error`, the ClusterIssuer was created outside of Helm. See [Helm Ownership Conflict](#helm-ownership-conflict) below.
 - **Domain not reachable**: Ensure your domain's DNS points to the ingress controller's external IP
 - **Rate limited**: Use `letsencrypt-staging` for testing to avoid production rate limits
 - **Challenge failed**: Check that port 80 is accessible for HTTP-01 challenges
 
+### Helm Ownership Conflict
+
+If you get an error like:
+```
+Error: UPGRADE FAILED: Unable to continue with update: ClusterIssuer "letsencrypt-prod" 
+in namespace "" exists and cannot be imported into the current release: 
+invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by"
+```
+
+This happens when:
+1. The ClusterIssuer was created manually with `kubectl apply`
+2. A Helm chart template tries to create/manage the same ClusterIssuer
+
+**Solution**: ClusterIssuers should be managed **outside** of this Helm chart:
+
+```bash
+# Option 1: Keep the manually created ClusterIssuer (recommended)
+# Simply don't include cluster-issuer templates in the chart
+# This overlay already follows this pattern - it references the ClusterIssuer
+# via annotation but doesn't create it
+
+# Option 2: If you have a local cluster-issuer template file, remove it
+rm helm/argo-stack/overlays/ingress-authz-overlay/templates/cluster-issuer*.yaml
+
+# Option 3: To adopt an existing resource into Helm (advanced)
+# Add Helm labels and annotations to the existing ClusterIssuer:
+kubectl annotate clusterissuer letsencrypt-prod \
+  meta.helm.sh/release-name=ingress-authz-overlay \
+  meta.helm.sh/release-namespace=argo-stack
+kubectl label clusterissuer letsencrypt-prod \
+  app.kubernetes.io/managed-by=Helm
+```
+
+**Note**: This chart intentionally does NOT include ClusterIssuer templates because ClusterIssuers are cluster-scoped resources that typically:
+- Pre-exist before deploying applications
+- Are shared across multiple releases
+- Should be managed separately from application charts
+
 ## Installation
 
 ### Prerequisites

From 652fbc3d888cc33abb489fb08d4d645f422e9aba Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 25 Nov 2025 01:36:13 +0000
Subject: [PATCH 6/6] Improve Helm ownership conflict documentation clarity

Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
---
 .../docs/authz-ingress-user-guide.md               | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
index 3470d6ea..4e57dbe7 100644
--- a/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
+++ b/helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md
@@ -276,9 +276,7 @@ Common issues:
 
 If you get an error like:
 ```
-Error: UPGRADE FAILED: Unable to continue with update: ClusterIssuer "letsencrypt-prod" 
-in namespace "" exists and cannot be imported into the current release: 
-invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by"
+Error: UPGRADE FAILED: Unable to continue with update: ClusterIssuer "letsencrypt-prod" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by"
 ```
 
 This happens when:
@@ -305,10 +303,12 @@ kubectl label clusterissuer letsencrypt-prod \
   app.kubernetes.io/managed-by=Helm
 ```
 
-**Note**: This chart intentionally does NOT include ClusterIssuer templates because ClusterIssuers are cluster-scoped resources that typically:
-- Pre-exist before deploying applications
-- Are shared across multiple releases
-- Should be managed separately from application charts
+**Why ClusterIssuers are managed separately**: ClusterIssuers are cluster-scoped resources that affect the entire cluster, not just one namespace. Including them in application-specific Helm charts causes conflicts when:
+- Multiple applications need the same ClusterIssuer
+- The ClusterIssuer already exists (created by a previous deployment or another chart)
+- Different teams deploy applications that reference the same issuer
+
+This chart references the ClusterIssuer via annotation (`cert-manager.io/cluster-issuer`) but leaves its lifecycle management to cluster administrators.
 
 ## Installation