[bitnami/etcd] bug: preupgrade will failed when startup etcd with one-way tls authentication

[chenraoCR]

Name and Version

bitnami/etcd:3.5.18

What architecture are you using?

None

What steps will reproduce the bug?

1. container.env

ALLOW_NONE_AUTHENTICATION = "yes"
ETCD_LISTEN_CLIENT_URLS = "https://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS = "https://etcd:2379"
ETCD_INITIAL_CLUSTER = "etcd=https://etcd:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS = "https://etcd:2379"
ETCD_NAME = "etcd"
ETCD_CERT_FILE = "/certs/tls.crt"
ETCD_KEY_FILE = "/certs/tls.key"

2. compose.yaml

services:
  etcd:
    image: docker.io/bitnami/etcd:3.5.18`
    restart: unless-stopped
    env_file: ./container.env
    #ports: [2379,2380]
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2G
    volumes:
      - etcd_data:/bitnami/etcd/data
      - ./certs:/certs

3. startup etcd and go into the container
4. run /opt/bitnami/scripts/etcd/preupgrade.sh will fail

What is the expected behavior?

preupgrade.sh should still succeed when startup etcd with one-way tls authentication

What do you see instead?

/opt/bitnami/scripts/etcd/preupgrade.sh failed

Additional information

same issue with #70554

[juan131]

Hi @chenraoCR

The preupgrade.sh script is intended to be executed on K8s environments (via K8s Job) while upgrading a chart release decreasing the number of etcd replicas. In this case, the script is in charge of removing obsolete replicas. Find more information in the link below:

• https://github.com/bitnami/charts/tree/main/bitnami/etcd#etcd-pre-upgrade-k8s-job-parameters

I tried to reproduce the issue installing the Bitnami etcd chart using the command and chart values below:

• Command:

helm install etcd oci://registry-1.docker.io/bitnamicharts/etcd -f values.yaml

• Values (as you can see TLS auth is enabled):

replicaCount: 3
auth:
  rbac:
    create: true
    rootPassword: some-root-password
  token:
    enabled: false
  client:
    secureTransport: true
    useAutoTLS: false
    existingSecret: etcd-client-crt
    certFilename: tls.crt
    certKeyFilename: tls.key
    caFilename: ca.crt
extraDeploy:
- |
  apiVersion: v1
  kind: Secret
  metadata:
    name: etcd-client-crt
    namespace: {{ include "common.names.namespace" . | quote }}
    labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
  type: kubernetes.io/tls
  data:
    {{- $ca := genCA "etcd-ca" 365 }}
    {{- $etcdFullname := include "common.names.fullname" . }}
    {{- $etcdHeadlessSvcName := printf "%s-headless" (include "common.names.fullname" .) | trunc 63 | trimSuffix "-" }}
    {{- $releaseNamespace := include "common.names.namespace" . }}
    {{- $clusterDomain := .Values.clusterDomain }}
    {{- $etcdAltNames := list (printf "*.%s.%s.svc.%s" $etcdHeadlessSvcName $releaseNamespace $clusterDomain) (printf "%s.%s.svc.%s" $etcdHeadlessSvcName $releaseNamespace $clusterDomain) $etcdHeadlessSvcName (printf "%s.%s.svc.%s" $etcdFullname $releaseNamespace $clusterDomain) $etcdFullname "127.0.0.1" "localhost" }}
    {{- $clientCert := genSignedCert $etcdFullname nil $etcdAltNames 365 $ca }}
    ca.crt: {{ $ca.Cert | b64enc | quote }}
    tls.crt: {{ $clientCert.Cert | b64enc | quote }}
    tls.key: {{ $clientCert.Key | b64enc | quote }}

Here some proofs that setup is working using TLS:

$ kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   1/1     Running   0          3m27s
etcd-1   1/1     Running   0          3m27s
etcd-2   1/1     Running   0          3m27s
$ kubectl logs etcd-0
(...)
{"level":"info","ts":"2025-02-26T09:54:06.051430Z","caller":"embed/etcd.go:762","msg":"starting with client TLS","tls-info":"cert = /bitnami/etcd/data/fixtures/client/cert.pem, key = /bitnami/etcd/data/fixtures/client/key.pem, client-cert=/bitnami/etcd/data/fixtures/client/cert.pem, client-key=/bitnami/etcd/data/fixtures/client/key.pem, trusted-ca = , client-cert-auth = false, crl-file = ","cipher-suites":[]}

Then, I upgrade the chart release decreasing the number of replicas:

helm upgrade etcd oci://registry-1.docker.io/bitnamicharts/etcd -f values.yaml --set replicaCount=2

And here you have the "pre-upgrade" logs:

$ kubectl logs job/etcd-pre-upgrade
(...)
etcd 10:26:48.69 INFO  ==> Current cluster members are: etcd-2,etcd-1,etcd-0
etcd 10:26:48.69 INFO  ==> Expected cluster members are: etcd-0,etcd-1
etcd 10:26:48.69 INFO  ==> Removing obsolete member etcd-2
Member 45a18acb10aa275e removed from cluster 9e98e654be1e22d7
etcd 10:26:48.89 INFO  ==> Pre-upgrade checks completed!
$ kubectl get pods
NAME                     READY   STATUS      RESTARTS   AGE
etcd-0                   1/1     Running     0          65s
etcd-1                   1/1     Running     0          2m9s
etcd-pre-upgrade-wtsq9   0/1     Completed   0          2m15s