Specifing docker --config option breaks docker buildx build command

[yankeguo]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I use different DOCKER_CONFIG directories to switch among different tenants of the same Container Registry.

Recently, I found that when I set the DOCKER_CONFIG directory parameter, whether through the --config parameter or the DOCKER_CONFIG environment variable, the docker buildx build command parameter parsing will be completely messed up, reporting that --platform, -t and other parameters cannot be recognized.

Expected behaviour

Command like

docker --config /path/to/custom-config-dir buildx build --platform linux/amd64 -t myimage1 -t myimage2 --build-arg APP_NAME=server --push .

should work

Actual behaviour

An error raised:

unknown flag: --platform
See 'docker --help'.

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Common Commands:
  run         Create and run a new container from an image
  exec        Execute a command in a running container
...

When I omit --config option, everything works.

Even if I set DOCKER_CONFIG environment variable, it will also fall.

Buildx version

github.com/docker/buildx v0.16.2-desktop.1 081c21b9e461293ae243a1ff813a680a4f5f8fb9

Docker info

Client:
 Version:    27.2.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     /Users/myusername/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     /Users/myusername/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/myusername/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/myusername/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/myusername/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/myusername/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/myusername/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/myusername/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/myusername/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     /Users/myusername/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.2.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.4-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.655GiB
 Name: docker-desktop
 ID: 9b936eee-5f78-458b-adf1-6c115f5a42e0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/myusername/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Builders list

-> % docker buildx ls
NAME/NODE           DRIVER/ENDPOINT     STATUS    BUILDKIT   PLATFORMS
default             docker                                   
 \_ default          \_ default         running   v0.15.2    linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64
desktop-linux*      docker                                   
 \_ desktop-linux    \_ desktop-linux   running   v0.15.2    linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64

Configuration

FROM alpine:3.20

ARG APP_NAME="server"

ENV TZ="Asia/Shanghai"

RUN apk add --no-cache tini ca-certificates tzdata curl && \
    ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
    echo $TZ > /etc/timezone

ENTRYPOINT [ "/sbin/tini", "--" ]

WORKDIR /data

ADD ${APP_NAME} /${APP_NAME}
RUN chmod +x /${APP_NAME}

ENV APP_NAME="${APP_NAME}"

CMD [ "sh", "-c", "exec /${APP_NAME}" ]

Build logs

No response

Additional info

No response

[tonistiigi]

@thaJeztah I can't reproduce but the error printed here("A self-sufficient runtime for containers") is coming from docker/cli

[thaJeztah]

I suspect the problem is in this part;

  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     /Users/myusername/.docker/cli-plugins/docker-buildx

CLI plugins are discovered in a number of locations; some "system-wide", and some "per user"

• /usr/[local/]lib/docker/cli-plugins
• /usr/[local/]libexec/docker/cli-plugins
• ~/.docker/cli-plugins/

The last one allows overriding the system-installed version, but I think the ~/.docker here is not hard-coded, but "config-directory". So using --config <custom directory> not only switches the location to look for the config-file, but also other content related to the config-directory (including those plugins, and likely contexts and state stored by plugins inside the config-directory).

Trying to mimic that situation;

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:cli
apk add -qq --no-cache js

docker info --format '{{json .ClientInfo.Plugins}}' | jq .
[
  {
    "SchemaVersion": "0.1.0",
    "Vendor": "Docker Inc.",
    "Version": "v0.17.1",
    "ShortDescription": "Docker Buildx",
    "Name": "buildx",
    "Path": "/usr/local/libexec/docker/cli-plugins/docker-buildx"
  },
  {
    "SchemaVersion": "0.1.0",
    "Vendor": "Docker Inc.",
    "Version": "v2.29.7",
    "ShortDescription": "Docker Compose",
    "Name": "compose",
    "Path": "/usr/local/libexec/docker/cli-plugins/docker-compose"
  }
]

Moving the plugins to ~/.docker/cli-plugins;

mkdir -p ~/.docker/cli-plugins/
mv usr/local/libexec/docker/cli-plugins/* ~/.docker/cli-plugins/

Without custom config-dir;

docker build --help
Start a build

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build
...

With a custom config-dir;

docker --config=/foo/ build --help
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/


Usage:  docker build [OPTIONS] PATH | URL | -

[thaJeztah]

This is somewhat of a known issue; there's various tickets (and backlog items) related to revamping storage locations (also in relation to rootless mode, which has some additional logic, and use of XDG directories). While it's a known issue, it's grown to be more problematic with the introduction of many plugins, and other parts of docker (ab)using ~/.docker for a growing list of purposes (which can be particularly problematic when that involves platform-specific content, and users (e.g.) bind-mount the config-directory into a container).

It's not a trivial thing to change though, as there's many tools that implemented these paths and logic, so at least it will require a long transition period; it will also involve changes in Docker Desktop (which uses this location for various purposes as well, including cli-plugins it ships).

Let me at least transfer this ticket to the CLI repository, which has the core parts of this logic, and link it from the internal backlog item(s) related to this.

[prathyk]

on my mac I get the following:

❯ docker --config ~/.docker/docker compose ls
docker: 'compose' is not a docker command.
See 'docker --help'