Fix 404 errors on ingress routes#105
Conversation
…mprove route defaults Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
"""
|
@copilot ```
|
The 404 from nginx could be caused by:
To debug, please check: # Verify authz-adapter is running
kubectl get pods -n security -l app=authz-adapter
# Test auth endpoint directly
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
curl -v http://authz-adapter.security.svc.cluster.local:8080/check
# Check the ExternalName proxy service
kubectl get svc -n argo-stack argo-stack-argo-workflows-server-proxy -o yamlIf auth is the issue, you can temporarily disable it by adding to the ingress annotations: nginx.ingress.kubernetes.io/enable-global-auth: "false" |
|
@copilot * authz-adapter is running and responds """ Verify authz-adapter is runningkubectl get pods -n security -l app=authz-adapter Test auth endpoint directlykubectl run -it --rm debug --image=curlimages/curl --restart=Never --
|
The debugging output confirms everything is working correctly:
The 401 response is correct behavior. When accessing routes via browser without authentication, NGINX should redirect to To test the full flow: # Should redirect to login (302)
curl -v https://calypr-demo.ddns.net/workflows
# With a valid token, should return the Argo UI
curl -v -H "Authorization: Bearer <token>" https://calypr-demo.ddns.net/workflowsAre you still seeing 404 errors when accessing the routes in a browser? If so, please share the NGINX ingress controller logs: kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=50 |
|
current routes: """ argo-events / github-repo-registrations-eventsource argo-stack / ingress-authz-applications argo-stack / ingress-authz-registrations argo-stack / ingress-authz-workflows calypr-api / ingress-authz-api calypr-tenants / ingress-authz-tenants |
|
@copilot It doesn't look like helm/argo-stack/overlays/ingress-authz-overlay/values.yaml ingressAuthzOverlay.routes is getting applied correctly. The rendered nginx config is below. There is only an """ configuration file /etc/nginx/nginx.conf:Configuration checksum: 6906240520238527922setup custom paths that do not require root accesspid /tmp/nginx/nginx.pid; daemon off; worker_processes 2; worker_rlimit_nofile 1073740792; worker_shutdown_timeout 240s ; events { } http { } stream { } configuration file /etc/nginx/mime.types:types { }""" |
The ingress resources ARE being created correctly (as shown in your previous Common causes:
Debug steps: # Check for TLS secret in argo-stack namespace
kubectl get secret calypr-demo-tls -n argo-stack
# Check NGINX controller logs for errors
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=100 | grep -i error
# Search nginx config for our host specifically
kubectl exec -n ingress-nginx -it $(kubectl get pods -n ingress-nginx -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}') -- grep -A 50 "calypr-demo.ddns.net" /etc/nginx/nginx.conf
# Force nginx reload
kubectl rollout restart deployment -n ingress-nginx ingress-nginx-controllerIf the TLS secret is missing, you can create it manually or check if cert-manager is issuing it: kubectl get certificate -A
kubectl get certificaterequest -A |
|
@copilot nginx logs state: """ |
This error indicates the NGINX controller can't access/watch ingresses in
kubectl get deployment -n ingress-nginx ingress-nginx-controller -o yaml | grep -A5 "args:"Look for
kubectl get clusterrolebinding -l app.kubernetes.io/name=ingress-nginx
kubectl describe clusterrole ingress-nginx
kubectl get ingressclass
kubectl get ingress -n argo-stack ingress-authz-workflows -o yaml | grep ingressClassNameQuick fix - if using namespace-scoped nginx, add helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.watchIngressWithoutClass=true |
|
@copilot """ """ events [] [] [create patch] |
The RBAC and namespace configuration look correct. The "could not find Ingress in local store" error usually indicates a sync/caching issue. Try these steps: # 1. Check IngressClass exists and matches
kubectl get ingressclass nginx -o yaml
# 2. Verify the ingress has correct ingressClassName
kubectl get ingress -n argo-stack ingress-authz-workflows -o jsonpath='{.spec.ingressClassName}'
# 3. Force controller to re-sync by restarting
kubectl rollout restart deployment -n ingress-nginx ingress-nginx-controller
# 4. Wait for ready, then check logs again
kubectl rollout status deployment -n ingress-nginx ingress-nginx-controller
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=20If the ingresses still aren't being loaded, try deleting and recreating them: helm uninstall ingress-authz-overlay -n argo-stack
helm upgrade --install ingress-authz-overlay helm/argo-stack/overlays/ingress-authz-overlay -n argo-stack |
There was a problem hiding this comment.
Pull request overview
This PR fixes 404 errors on ingress routes by addressing two main issues: (1) ArgoCD backend protocol misconfiguration (HTTP vs HTTPS), and (2) routes without explicit enabled: true being skipped by the template. The changes add support for backend protocol configuration, proxy timeout annotations, and fix the template's enabled flag handling to default to true.
Key Changes:
- Added
backendProtocolannotation support for HTTPS/gRPC backends with ArgoCD configured to use HTTPS - Fixed template logic to default route
enabledflag totrueinstead of requiring explicit configuration - Added NGINX ingress controller configuration to allow critical-risk annotations like
auth-snippet
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| helm/argo-stack/values.yaml | Adds backendProtocol: HTTPS comment and configuration for ArgoCD route |
| helm/argo-stack/overlays/ingress-authz-overlay/values.yaml | Updates namespaces for routes and adds backendProtocol: HTTPS for ArgoCD |
| helm/argo-stack/overlays/ingress-authz-overlay/values-ingress-nginx.yaml | New file configuring NGINX to allow snippet annotations at Critical risk level |
| helm/argo-stack/overlays/ingress-authz-overlay/templates/ingress-authz.yaml | Adds backend protocol, proxy timeout annotations support; fixes enabled default; hardens ingressClassName |
| helm/argo-stack/overlays/ingress-authz-overlay/templates/externalname-services.yaml | Fixes enabled flag default to true |
| helm/argo-stack/overlays/ingress-authz-overlay/docs/authz-ingress-user-guide.md | Adds 404 troubleshooting section with detailed debugging steps |
| helm/argo-stack/overlays/ingress-authz-overlay/README.md | Documents route configuration options including backend protocol |
| Makefile | Adds TLS secret creation for all namespaces; comments out PUBLIC_IP logic; adds ingress-nginx configuration |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,4 @@ | |||
| controller: | |||
| config: | |||
There was a problem hiding this comment.
Enabling snippet annotations with allow-snippet-annotations: true and annotations-risk-level: Critical creates a security risk. Snippet annotations allow arbitrary NGINX configuration code injection and should only be used in trusted environments. Consider documenting this security implication in a comment or the README, and ensure this configuration is intentional for the deployment environment.
| config: | |
| config: | |
| # WARNING: Enabling snippet annotations with `allow-snippet-annotations: true` and | |
| # `annotations-risk-level: Critical` allows arbitrary NGINX configuration code injection | |
| # via Ingress annotations. This creates a significant security risk and should ONLY be | |
| # used in trusted environments where Ingress resources are tightly controlled. | |
| # See: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#snippet-annotations |
| {{- end }} | ||
| spec: | ||
| ingressClassName: {{ $config.ingressClassName | default "nginx" | quote }} | ||
| ingressClassName: "nginx" |
There was a problem hiding this comment.
The ingressClassName is now hardcoded to \"nginx\" instead of using the configurable $config.ingressClassName | default \"nginx\". This reduces flexibility and is inconsistent with the values.yaml which defines ingressClassName: nginx. Revert to using the template variable: ingressClassName: {{ $config.ingressClassName | default \"nginx\" | quote }}.
| ingressClassName: "nginx" | |
| ingressClassName: {{ $config.ingressClassName | default "nginx" | quote }} |
| ``` | ||
| - Verify backend protocol settings (ArgoCD requires HTTPS): | ||
| ```bash | ||
| kubectl get ingress ingress-authz-applications -n argo-stack -o yaml | grep backend-protocol |
There was a problem hiding this comment.
The namespace in this command is incorrect. Based on the values.yaml changes, the applications ingress is created in the argocd namespace (line 122 of values.yaml), not argo-stack. Update the command to: kubectl get ingress ingress-authz-applications -n argocd -o yaml | grep backend-protocol.
| kubectl get ingress ingress-authz-applications -n argo-stack -o yaml | grep backend-protocol | |
| kubectl get ingress ingress-authz-applications -n argocd -o yaml | grep backend-protocol |
| ```bash | ||
| kubectl get svc -n argo-stack -l app.kubernetes.io/component=externalname-proxy |
There was a problem hiding this comment.
This command only checks for ExternalName services in the argo-stack namespace. However, based on the values.yaml configuration, proxy services are created in the ingress namespace where each route is defined (e.g., argocd, argo-workflows, argo-events). Consider adding a note that ExternalName services are created in each route's namespace, or use -A to check all namespaces: kubectl get svc -A -l app.kubernetes.io/component=externalname-proxy.
| ```bash | |
| kubectl get svc -n argo-stack -l app.kubernetes.io/component=externalname-proxy | |
| > **Note:** ExternalName proxy services are created in the namespace for each route (e.g., `argocd`, `argo-workflows`, `argo-events`), not just in `argo-stack`. | |
| ```bash | |
| kubectl get svc -A -l app.kubernetes.io/component=externalname-proxy |
| # @if [ -n "${PUBLIC_IP}" ]; then \ | ||
| # echo "➡️ Assigning external IP: ${PUBLIC_IP}"; \ | ||
| # kubectl patch svc ingress-nginx-controller -n ingress-nginx -p "{\"spec\": {\"type\": \"NodePort\", \"externalIPs\": [\"${PUBLIC_IP}\"]}}"; \ | ||
| # else \ | ||
| # echo "⚠️ PUBLIC_IP not set, skipping external IP assignment"; \ | ||
| # fi |
There was a problem hiding this comment.
Large blocks of commented-out code should be removed rather than committed. If this PUBLIC_IP assignment logic might be needed in the future, document why it was disabled in a comment and consider removing the code block, or explain the rationale for keeping it commented.
All ingress routes (
/workflows,/applications,/registrations,/api,/tenants) were returning 404 errors despite backend services being reachable directly.Root Causes & Fixes
backendProtocol: HTTPSannotation for/applicationsrouteenabled: truewere skipped. Changed template to defaultenabledtotrueChanges
backendProtocolannotation support for HTTPS/GRPC backendsbackendProtocol: HTTPSby defaultproxyConnectTimeout,proxyReadTimeout,proxySendTimeout)enabledflag to defaulttruein bothingress-authz.yamlandexternalname-services.yamlUsage
After upgrade, verify:
kubectl get ingress ingress-authz-applications -n argo-stack -o yaml | grep backend-protocol💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.