In repo tags 7.2 and up in zabbix-server HashiCorp Vault params collide with DBUser, DBPassword

[Contik]

SUMMARY

Hi!

I believe that in docker-compose_v3_alpine_pgsql_latest.yaml (possibly in all other Compose files) this repo somewhat recently introduced a regression. When I use HashiCorp Vault as indicated by setting environment variables VAULT_TOKEN, ZBX_VAULT, ZBX_VAULTDBPATH and ZBX_VAULTURL as of roughly the 7.2.x tags docker-entrypoint.sh lines 397 and 398 will indiscriminately Bash export variables ZBX_DB_USER and ZBX_DB_PASSWORD.

The Docker Compose zabbix-server service container will pick this up in its /etc/zabbix/zabbix_server.conf which does an Include=/etc/zabbix/zabbix_server_db.conf which in turn renders both DBUser=${ZBX_DB_USER} and DBPassword=${ZBX_DB_PASSWORD}.

If either DBUser or DBPassword is set Zabbix --entrypoint "/usr/bin/docker-entrypoint.sh" with params "/usr/sbin/zabbix_server", "--foreground", "-c", "/etc/zabbix/zabbix_server.conf" will politely exit.

There seems to be no obvious way to use HashiCorp Vault now which used to work fine with repo version 7.0 and up.

OS / ENVIRONMENT / Used docker-compose files

This is running docker-compose_v3_alpine_pgsql_latest.yaml on an Arch Linux rolling release machine.

Docker version:

# docker version
Client:
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.23.5
 Git commit:        9f9e405801
 Built:             Tue Jan 28 21:43:20 2025
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.23.1
  Git commit:       41ca978a0a
  Built:            Wed Sep 25 14:43:43 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.0.2
  GitCommit:        c507a0257ea6462fbd6f5ba4f5c74facb04021f4.m
 runc:
  Version:          1.2.5
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Compose version:

# docker compose version
Docker Compose version 2.33.1

Image tags and manifest hashes:

# docker image ls -a
REPOSITORY                            TAG                 IMAGE ID       CREATED        SIZE
postgres                              16-alpine           1e8624fe3776   9 days ago     275MB
zabbix/zabbix-web-nginx-pgsql         alpine-7.2-latest   4df5985dd0e6   9 days ago     228MB
zabbix/zabbix-server-pgsql            alpine-7.2-latest   236a5d6fd394   9 days ago     68.9MB
busybox                               latest              31311c5853a2   4 months ago   4.27MB

CONFIGURATION

In both env_vars/.env_web and env_vars/.env_srv I set:

VAULT_TOKEN=my-vault-token
ZBX_VAULT=HashiCorp
ZBX_VAULTDBPATH=kv/secret/zabbix
ZBX_VAULTURL=https://fully.qualified.domain.name

Until recently I was running with repo commit hash 77d0151 (tag 7.0.9) which worked fine. What I believe is happening is that back then docker-entrypoint.sh from commit hash 77d0151 used to run an update_zbx_config() Bash function where it would conditionally set DBUser and DBPassword to empty strings when Vault parameters had non-null values:

if [ -n "${ZBX_VAULTDBPATH}" ]; then
    update_config_var $ZBX_CONFIG "DBUser"
    update_config_var $ZBX_CONFIG "DBPassword"
else
    ...
fi

This allowed Zabbix to start just fine.

Somewhere around the 7.2.x tags - I believe specifically as of commit hash 688ed8b - this no longer works. Now both DBUser=${ZBX_DB_USER} and DBPassword=${ZBX_DB_PASSWORD} persist with non-null values and Zabbix exits on container start when it also has Vault params. Commit hash 688ed8b is contained in both the 7.2.2 and the 7.2.3 tags.

STEPS TO REPRODUCE

First without HashiCorp Vault params to confirm we're not forgetting anything.

1. Clone repo https://github.com/zabbix/zabbix-docker, for example into /opt/git/github.com/zabbix/zabbix-docker/branches/latest:

   export UPSTREAM_REPO_DIR='/opt/git/github.com/zabbix/zabbix-docker/branches/latest'
   mkdir -p "${UPSTREAM_REPO_DIR}"
   git -C "${UPSTREAM_REPO_DIR}" clone 'https://github.com/zabbix/zabbix-docker' .
   

2. Get affected version, for example the newest 7.2.x tag which at this time is 7.2.3:

   git -C "${UPSTREAM_REPO_DIR}" checkout "$(git --no-pager -C "${UPSTREAM_REPO_DIR}" tag -l --sort -version:refname | grep -Fi -- '7.2.' | head -n 1)"
   

3. cd into repo so Docker Compose can render a suitable --project-name from your current working directory:

   cd "${UPSTREAM_REPO_DIR}"
   export UPSTREAM_COMPOSE_FILE="${UPSTREAM_REPO_DIR%/}"'/docker-compose_v3_alpine_pgsql_latest.yaml'
   

4. (Optional) Get your bearing, Docker Compose will render this setup like this pastebin snippet at http://0x0.st/8T9S.txt. Snippet will auto-delete a month from now on Tuesday, March 25, 2025, 08:34:51 UTC:

   docker compose --file "${UPSTREAM_COMPOSE_FILE}" config
   

5. Pull images

   docker compose --file "${UPSTREAM_COMPOSE_FILE}" pull
   

   At this time this will pull the following image tags and manifest hashes:

   # docker image ls -a
   REPOSITORY                      TAG                 IMAGE ID       CREATED        SIZE
   postgres                        16-alpine           869ba6a9adbd   2 days ago     275MB
   zabbix/zabbix-web-nginx-pgsql   alpine-7.2-latest   4df5985dd0e6   9 days ago     228MB
   zabbix/zabbix-server-pgsql      alpine-7.2-latest   236a5d6fd394   9 days ago     68.9MB
   busybox                         latest              31311c5853a2   4 months ago   4.27MB
   

6. Create directory structure that Compose expects:

   mkdir -p "${UPSTREAM_REPO_DIR}"'/zbx_env/etc/ssl/nginx'
   

7. For an SSL-secured connection get a dhparam file or generate one yourself. The lazy way uses for example Mozilla's file:

   curl \
      --location \
      --silent \
      'https://ssl-config.mozilla.org/ffdhe2048.txt' \
      --output \
      "${UPSTREAM_REPO_DIR}"'/zbx_env/etc/ssl/nginx/dhparam.pem'
   

8. Get SSL cert files into "${UPSTREAM_REPO_DIR}"'/env_vars'

       .
       ├── env_vars
   --> │   ├── .ZBX_DB_CA_FILE
   --> │   ├── .ZBX_DB_CERT_FILE
   --> │   └── .ZBX_DB_KEY_FILE
       └── zbx_env
           └── etc
               └── ssl
                   └── nginx
                       └── dhparam.pem
   

9. Copy *CERT_FILE and *KEY_FILE over to "${UPSTREAM_REPO_DIR}"'/zbx_env/etc/ssl/nginx' as ssl.crt and ssl.key:

       .
       ├── env_vars
       │   ├── .ZBX_DB_CA_FILE
       │   ├── .ZBX_DB_CERT_FILE
       │   └── .ZBX_DB_KEY_FILE
       └── zbx_env
           └── etc
               └── ssl
                   └── nginx
                       ├── dhparam.pem
   -->                 ├── ssl.crt
   -->                 └── ssl.key
   

10. Start containers

    docker compose --file "${UPSTREAM_COMPOSE_FILE}" up --detach
    

11. Visit your new empty Zabbix instance at whatever your fully.qualified.domain.name is:
    

Stop your instance, redo with HashiCorp Vault params.

1. To both env_vars/.env_web and env_vars/.env_srv add the same four Vault parameters that work fine with commit hash 77d0151 (tag 7.0.9). You've confirmed they are able to retrieve the zabbix secret with both its username and password key and both keys have their value set to zabbix as per env_vars/.POSTGRES_PASSWORD and env_vars/.POSTGRES_USER:

   VAULT_TOKEN=my-vault-token
   ZBX_VAULT=HashiCorp
   ZBX_VAULTDBPATH=kv/secret/zabbix
   ZBX_VAULTURL=https://fully.qualified.domain.name
   

2. (Optional) Render the updated Docker Compose config to confirm that your added parameters are picked up correctly as in this pastebin snippet at http://0x0.st/8T9I.txt. Snippet will auto-delete a month from now on Tuesday, March 25, 2025, 09:22:26 UTC.

   docker compose --file "${UPSTREAM_COMPOSE_FILE}" config
   

   This will create a diff to your first version like so where both the zabbix-server Docker Compose service and the zabbix-web-nginx-pgsql service now contain additional env vars:

   --- <unnamed>
   +++ <unnamed>
   @@ -111,9 +111,13 @@
          POSTGRES_DB: zabbix
          POSTGRES_PASSWORD_FILE: /run/secrets/POSTGRES_PASSWORD
          POSTGRES_USER_FILE: /run/secrets/POSTGRES_USER
   +      VAULT_TOKEN: my-vault-token
          ZBX_ENABLE_SNMP_TRAPS: "true"
          ZBX_JAVAGATEWAY: zabbix-java-gateway
          ZBX_STARTJAVAPOLLERS: "5"
   +      ZBX_VAULT: HashiCorp
   +      ZBX_VAULTDBPATH: kv/secret/zabbix
   +      ZBX_VAULTURL: https://fully.qualified.domain.name
        image: zabbix/zabbix-server-pgsql:alpine-7.2-latest
        init: true
        labels:
   @@ -251,8 +255,12 @@
          POSTGRES_DB: zabbix
          POSTGRES_PASSWORD_FILE: /run/secrets/POSTGRES_PASSWORD
          POSTGRES_USER_FILE: /run/secrets/POSTGRES_USER
   +      VAULT_TOKEN: my-vault-token
          ZBX_SERVER_HOST: zabbix-server
          ZBX_SERVER_NAME: Composed installation
   +      ZBX_VAULT: HashiCorp
   +      ZBX_VAULTDBPATH: kv/secret/zabbix
   +      ZBX_VAULTURL: https://fully.qualified.domain.name
        healthcheck:
          test:
            - CMD

3. Start containers

   docker compose --file "${UPSTREAM_COMPOSE_FILE}" up --detach
   

4. Notice how the zabbix-server-1 container exits:

   cannot initialize database credentials from vault: "DBUser" configuration parameter cannot be used when "VaultDBPath" is defined
   

Notice that the zabbix-web-nginx-pgsql Compose service doesn't have the same issue as it uses a different mechanism to unset database credentials.

EXPECTED RESULTS

Zabbix server container does either one of the two:

1. Automatically unset DBUser and DBPassword for me when Vault params are present
2. Allow me to unset POSTGRES_PASSWORD and POSTGRES_USER (or their _FILE counterparts)

ACTUAL RESULTS

Zabbix server container politely points out that DBUser and DBPassword must be unset or null when Vault params are present.

My quick and dirty workaround is to use docker-entrypoint.sh and to comment out lines 397 and 398 like so:

--- <unnamed>
+++ <unnamed>
@@ -394,8 +394,8 @@
 
     export ZBX_DB_NAME="${DB_SERVER_DBNAME}"
     export ZBX_DB_SCHEMA="${DB_SERVER_SCHEMA}"
-    export ZBX_DB_USER="${DB_SERVER_ZBX_USER}"
-    export ZBX_DB_PASSWORD="${DB_SERVER_ZBX_PASS}"
+    # export ZBX_DB_USER="${DB_SERVER_ZBX_USER}"
+    # export ZBX_DB_PASSWORD="${DB_SERVER_ZBX_PASS}"
 
     : ${ZBX_ENABLE_SNMP_TRAPS:="false"}
     [[ "${ZBX_ENABLE_SNMP_TRAPS,,}" == "true" ]] && export ZBX_STARTSNMPTRAPPER=1

I then place a compose.override.yaml file in the Git repo's root dir like so:

services:
  zabbix-server:
    volumes:
      - "/opt/git/github.com/zabbix/zabbix-docker/branches/latest/Dockerfiles/server-pgsql/alpine/docker-entrypoint.sh:/usr/bin/docker-entrypoint.sh:ro"

When I next render the resulting config (with --file compose.override.yaml) I confirm that a new bind mount overwrites the container's docker-entrypoint.sh file:

docker compose --file "${UPSTREAM_COMPOSE_FILE}" --file compose.override.yaml config

This picks up my own docker-entrypoint.sh as a bind mount into the container:

--- <unnamed>
+++ <unnamed>
@@ -234,6 +234,12 @@
       - type: bind
         source: /etc/timezone
         target: /etc/timezone
+        read_only: true
+        bind:
+          create_host_path: true
+      - type: bind
+        source: /opt/git/github.com/zabbix/zabbix-docker/branches/latest/Dockerfiles/server-pgsql/alpine/docker-entrypoint.sh
+        target: /usr/bin/docker-entrypoint.sh
         read_only: true
         bind:
           create_host_path: true

Is there a cleaner/better way to use HashiCorp Vault params with repo tags 7.2 and up?

Thanks!