Add DNS-01 challenge debugging guide, manual certificate installation, and NodePort external IP configuration to troubleshooting docs#102
Conversation
|
Read most up to date branch How to debug dns01 flow? We still have self signed certificates! |
…rror fixes Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
The error Quick diagnostic steps:
dig _acme-challenge.calypr-demo.ddns.net CNAME +short
# Should return: xxx.auth.acme-dns.io
FULLDOMAIN=$(kubectl get secret acme-dns-credentials -n cert-manager -o jsonpath='{.data.acmedns\.json}' | base64 -d | jq -r '."calypr-demo.ddns.net".fulldomain')
dig @auth.acme-dns.io $FULLDOMAIN TXT +short
kubectl logs -n cert-manager -l app=cert-manager -f | grep -i "dns\|propagation"If CNAME is missing, add it in No-IP.com:
See comprehensive debugging guide in |
|
I've manually retrieved my certificate. It is stored at """ |
To install your manually retrieved certificate: # Create the TLS secret from your certificate files
kubectl create secret tls calypr-demo-tls \
-n argo-stack \
--cert=/etc/letsencrypt/live/calypr-demo.ddns.net/fullchain.pem \
--key=/etc/letsencrypt/live/calypr-demo.ddns.net/privkey.pem
# If the secret already exists, delete it first:
# kubectl delete secret calypr-demo-tls -n argo-stackStop cert-manager from managing this certificate: # Remove cert-manager annotation from ingress
kubectl annotate ingress ingress-authz-workflows -n argo-stack \
cert-manager.io/cluster-issuer-
# Delete the Certificate resource (prevents cert-manager from overwriting)
kubectl delete certificate calypr-demo-tls -n argo-stackYour ingress will now use the manual certificate. Note: You'll need to manually renew it before expiration (Let's Encrypt certs are valid for 90 days). To re-enable automated cert-manager management later:
|
|
@copilot update documentation with instructions how to install a manual certificate |
Co-authored-by: bwalsh <47808+bwalsh@users.noreply.github.com>
Added comprehensive manual certificate installation guide to The new section includes:
See "Installing a Manual Certificate" section in the troubleshooting guide. |
|
@copilot What permission should be set on /etc/letsencrypt/live/calypr-demo.ddns.net/fullchain.pem
|
For Let's Encrypt certificate files, use these permissions: # Certificate files (public) - readable by all
chmod 644 /etc/letsencrypt/live/calypr-demo.ddns.net/fullchain.pem
# Private key - readable only by owner
chmod 600 /etc/letsencrypt/live/calypr-demo.ddns.net/privkey.pem
# Ensure proper ownership (usually root)
chown root:root /etc/letsencrypt/live/calypr-demo.ddns.net/*.pemSecurity Note: The private key ( When copying to create the Kubernetes secret, ensure you have read permissions. If running # Option 1: Use sudo
sudo kubectl create secret tls calypr-demo-tls -n argo-stack \
--cert=/etc/letsencrypt/live/calypr-demo.ddns.net/fullchain.pem \
--key=/etc/letsencrypt/live/calypr-demo.ddns.net/privkey.pem
# Option 2: Copy to accessible location first
sudo cp /etc/letsencrypt/live/calypr-demo.ddns.net/fullchain.pem /tmp/
sudo cp /etc/letsencrypt/live/calypr-demo.ddns.net/privkey.pem /tmp/
sudo chmod 644 /tmp/fullchain.pem /tmp/privkey.pem
kubectl create secret tls calypr-demo-tls -n argo-stack \
--cert=/tmp/fullchain.pem --key=/tmp/privkey.pem
rm /tmp/fullchain.pem /tmp/privkey.pem |
|
@copilot The secret looks good """ This last step is failing: """ Test HTTPS connection$ curl -vI https://calypr-demo.ddns.net/workflows 2>&1
curl failed to verify the legitimacy of the server and therefore could not """ """
|
The certificate is installed correctly (valid Let's Encrypt cert expires Feb 23, 2026). The issue is DNS resolution - Diagnose: # Check what IP the domain resolves to
nslookup calypr-demo.ddns.net
dig calypr-demo.ddns.net +short
# Get your ingress controller's external IP
kubectl get svc -n ingress-nginx ingress-nginx-controllerThe domain should resolve to your ingress IP, not 127.0.0.1. Fix options:
grep calypr-demo /etc/hosts
# If it shows 127.0.0.1, remove or update that line
# Get ingress IP
INGRESS_IP=$(kubectl get svc ingress-nginx-controller -n ingress-nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
# Test directly (override DNS)
curl -vI --resolve calypr-demo.ddns.net:443:$INGRESS_IP https://calypr-demo.ddns.net/workflowsOnce DNS points to the correct IP, the certificate will work. The "self-signed certificate" error is from a different service running on localhost, not your Kubernetes ingress. |
|
@copilot Still getting self signed """ Non-authoritative answer: ++ dig calypr-demo.ddns.net +short curl failed to verify the legitimacy of the server and therefore could not |
DNS is correct now. The issue is that ingress-nginx is type Verify the certificate is loaded: # Check if ingress-nginx sees the TLS secret
kubectl get secret calypr-demo-tls -n argo-stack -o yaml | grep -A1 "tls.crt\|tls.key"
# Check ingress-nginx logs for certificate loading
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=50 | grep -i "tls\|certificate\|calypr-demo"
# Force reload of ingress config
kubectl delete pod -n ingress-nginx -l app.kubernetes.io/component=controllerTest via NodePort directly: # Access via the actual NodePort (bypasses any port forwarding issues)
curl -vI --resolve calypr-demo.ddns.net:30443:100.22.124.96 https://calypr-demo.ddns.net:30443/workflowsIf NodePort works but port 443 doesn't: You need to forward port 443 to NodePort 30443 on the node. Options: # Option 1: Use iptables (on the node with IP 100.22.124.96)
sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 30443
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 30080
# Option 2: Switch to LoadBalancer (recommended)
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"LoadBalancer"}}' |
User asked how to configure external IP addresses for NodePort ingress in comment #3577836856, how to debug DNS-01 flow with "DNS record not yet propagated" errors in comment #3577905433, and how to install a manual certificate in comment #3578039890.
Changes
NodePort External IP Configuration
DNS-01 Challenge Debugging
Manual Certificate Installation
Example Configuration
NodePort with MetalLB (assigns external IP automatically)
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"LoadBalancer"}}'NodePort with Manual externalIPs
DNS-01 Quick Diagnostic
Manual Certificate Installation
Documentation-only changes - no code modifications.
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.