Contour occasionally gets stuck in a non-ready state and fails to start the XDS server

[tsaarni]

Contour occasionally gets stuck in a non-ready state and fails to start the XDS server. This issue is uncommon and occurs only under the following conditions:

• --watch-namespaces flag is set to monitor a single namespace.
• That namespace has only "unrelated" secrets that Contour does not process.
• The stuck Contour instance is the follower (not leader).

To reproduce this issue, follow these steps:

1. Deploy contour:

$ kubectl apply -f https://projectcontour.io/quickstart/contour.yaml

2. Create a nearly empty namespace with one generic secret:

$ kubectl create ns almost-empty
$ kubectl -n almost-empty create secret generic my-secret --from-literal=secret=value

3. Configure Contour to watch only the almost-empty namespace:

$ kubectl patch deployment contour -n projectcontour --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--watch-namespaces=almost-empty"}]'

4. Restart both Contour instances by scaling down and back up:

$ kubectl -n projectcontour scale deployment --replicas=0 contour
$ kubectl -n projectcontour scale deployment --replicas=2 contour

5. Verify that one Contour instance remains stuck in a non-ready state:

$ kubectl -n projectcontour get pod -l app=contour
NAME                      READY   STATUS    RESTARTS   AGE
contour-db59b775d-9k4wl   1/1     Running   0          4m17s
contour-db59b775d-tnqxm   0/1     Running   0          4m17s

The problem is a race condition, and its likelihood decreases if more resources (or no resources) are created in the watched namespace.

[github-actions]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack