etcd unstable in rancher+k8s 1.19.8

[yangtian9999]

use default apisix helm chart to install, whole components deployment in rancher+k8s 1.19.8

NAME READY STATUS RESTARTS AGE
apisix-8c6fdd4bb-9hpx7 1/1 Running 4 28m
apisix-8c6fdd4bb-gtq9b 1/1 Running 0 28m
apisix-dashboard-5c74b68655-8ttxm 1/1 Running 4 28m
apisix-etcd-0 1/1 Running 9 28m
apisix-etcd-1 0/1 CrashLoopBackOff 5 28m

Name: apisix-etcd-0
Namespace: apisix
Priority: 0
Node: xx.xx.xx.xx/xx.xx.xx.xx
Start Time: Tue, 24 Aug 2021 14:54:33 +0800
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
controller-revision-hash=apisix-etcd-6c67d8888c
helm.sh/chart=etcd-6.2.6
statefulset.kubernetes.io/pod-name=apisix-etcd-0
Annotations: cni.projectcalico.org/podIP: 10.42.208.237/32
cni.projectcalico.org/podIPs: 10.42.208.237/32
Status: Running
IP: 10.42.208.237
IPs:
IP: 10.42.208.237
Controlled By: StatefulSet/apisix-etcd
Containers:
etcd:
Container ID: docker://545d77109b975b5f79cbf0d31ae4c8e6c9d8720968135d84e7a5245d95df023c
Image: docker.io/bitnami/etcd:3.4.16-debian-10-r14
Image ID: docker-pullable://bitnami/etcd@sha256:ef2d499749c634588f7d281dd70cc1fb2514d57f6d42308c0fb0f2c8ca55bea4
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 24 Aug 2021 15:18:15 +0800
Finished: Tue, 24 Aug 2021 15:19:18 +0800
Ready: False
Restart Count: 8
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: apisix-etcd-0 (v1:metadata.name)
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_ADVERTISE_CLIENT_URLS: http://$(MY_POD_NAME).apisix-etcd-headless.apisix.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://$(MY_POD_NAME).apisix-etcd-headless.apisix.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER: apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etc d-1=http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN: apisix-etcd-headless.apisix.svc.cluster.local
Mounts:
/bitnami/etcd from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-hc8n7 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-apisix-etcd-0
ReadOnly: false
default-token-hc8n7:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-hc8n7
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning FailedScheduling 25m default-scheduler 0/6 nodes are available: 6 pod has unbound immediate PersistentVolum eClaims.
Warning FailedScheduling 25m default-scheduler 0/6 nodes are available: 6 pod has unbound immediate PersistentVolum eClaims.
Normal Scheduled 25m default-scheduler Successfully assigned apisix/apisix-etcd-0 to xx.xx.xx.xx
Normal Pulling 25m kubelet Pulling image "docker.io/bitnami/etcd:3.4.16-debian-10-r14"
Normal Pulled 24m kubelet Successfully pulled image "docker.io/bitnami/etcd:3.4.16-debian-10-r 14" in 1m2.17843604s
Warning Unhealthy 7m11s kubelet Liveness probe errored: rpc error: code = Unknown desc = container n ot running (81e29f66f5f9edd86d23e2f1317ec8809d9d0dc202125fa0c7236cbc3b972a23)
Normal Created 6m57s (x6 over 24m) kubelet Created container etcd
Normal Started 6m57s (x6 over 24m) kubelet Started container etcd
Normal Pulled 5m16s (x6 over 24m) kubelet Container image "docker.io/bitnami/etcd:3.4.16-debian-10-r14" alread y present on machine
Warning BackOff 11s (x26 over 23m) kubelet Back-off restarting failed container

[yangtian9999]

because of the unstable etcd. we could not submit the route or upstream by curl apisix admin

[tokers]

@yangtian9999 It seems that this is related to the PVC, I don't know the default storage class of Rancher, maybe you can specify the storage class explicitly and see what happens.

[yangtian9999]

@tokers Please check these, Thanks.

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storage-nfs (default) storage.pri/nfs Delete Immediate false 8d

---

pvc-8fa65076-83d6-4f84-ae43-c461ba727e46 8Gi RWO Delete Bound apisix/data-apisix-etcd-1 storage-nfs 19h
pvc-9b1de072-8831-4847-83e8-0d0d05e5c625 8Gi RWO Delete Bound apisix/data-apisix-etcd-0 storage-nfs 19h

---

apisix data-apisix-etcd-0 Bound pvc-9b1de072-8831-4847-83e8-0d0d05e5c625 8Gi RWO storage-nfs 19h
apisix data-apisix-etcd-1 Bound pvc-8fa65076-83d6-4f84-ae43-c461ba727e46 8Gi RWO storage-nfs 19h

---

Name: pvc-8fa65076-83d6-4f84-ae43-c461ba727e46
Labels:
Annotations: pv.kubernetes.io/provisioned-by: storage.pri/nfs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: storage-nfs
Status: Bound
Claim: apisix/data-apisix-etcd-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 8Gi
Node Affinity:
Message:
Source:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: xx.xx.xx.xx
Path: /nfs/data/apisix-data-apisix-etcd-1-pvc-8fa65076-83d6-4f84-ae43-c461ba727e46
ReadOnly: false
Events:

---

Name: pvc-9b1de072-8831-4847-83e8-0d0d05e5c625
Labels:
Annotations: pv.kubernetes.io/provisioned-by: storage.pri/nfs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: storage-nfs
Status: Bound
Claim: apisix/data-apisix-etcd-0
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 8Gi
Node Affinity:
Message:
Source:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: xx.xx.xx.xx
Path: /nfs/data/apisix-data-apisix-etcd-0-pvc-9b1de072-8831-4847-83e8-0d0d05e5c625
ReadOnly: false
Events:

---

Name: data-apisix-etcd-0
Namespace: apisix
StorageClass: storage-nfs
Status: Bound
Volume: pvc-9b1de072-8831-4847-83e8-0d0d05e5c625
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/name=etcd
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: storage.pri/nfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 8Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: apisix-etcd-0
Events:

---

Name: data-apisix-etcd-1
Namespace: apisix
StorageClass: storage-nfs
Status: Bound
Volume: pvc-8fa65076-83d6-4f84-ae43-c461ba727e46
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/name=etcd
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: storage.pri/nfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 8Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: apisix-etcd-1
Events:

[tokers]

See https://kubernetes.io/docs/concepts/storage/storage-classes/#nfs for more details, you can search more about nfs and learn some constraints about it.

[yangtian9999]

apisix etcd pvc bounded to default storage class successfully.
We deployed some apps with nfs provided pvc normally.