Multiarch: Does buildah push need an --arch flag?

[anthr76]

Hi all. Throughout the week I've been experimenting with getting multi-arch builds going on GitLab CI. I've overcame most of the major hurdles, though stuck on one..

Consider this Gitlab job:

.buildah native:
  image: quay.io/buildah/stable:v1.21.0
  extends:
    - .container build
  services:
    - docker:dind
  variables:
    FF_ENABLE_BASH_EXIT_CODE_CHECK: "true"
  before_script:
    - export CONTAINER_HOST=tcp://docker:2375/
    - echo "Building ${GL_BUILD_TAG} ${QUAY_BUILD_TAG} using ${SCRIPT_DIR}/${SCRIPT_NAME}"
    - mkdir -p /run/containers/$UID/
    - echo "$DOCKERCFG" > /run/containers/$UID/auth.json
    - export fed_ver=$(echo $FGC | tr -d f)
    - mkdir -p /var/cache/dnf/$fed_ver
    - dnf install -y podman
    - sudo podman run --rm --privileged docker.io/multiarch/qemu-user-static --reset -p yes
    - buildah manifest create --arch $ARCH $QUAY_BUILD_TAG
    - buildah manifest create --arch $ARCH $GL_BUILD_TAG
  script:
    - bash -e $SCRIPT_DIR/$SCRIPT_NAME
    - export CONTAINER=$(buildah containers --format "{{.ContainerName}}")
    - buildah config --arch $ARCH $CONTAINER
    - buildah commit --manifest $GL_BUILD_TAG $CONTAINER
    - buildah commit --manifest $QUAY_BUILD_TAG $CONTAINER
    - buildah push -f oci --tls-verify=false $GL_BUILD_TAG
    - buildah push $QUAY_BUILD_TAG
    - mkdir -p build
    - buildah push -f docker $GL_BUILD_TAG oci:build/image
  parallel:
    matrix:
      - ARCH:
        - amd64
        - arm64
  cache:
    paths:
      - /var/cache/dnf
  artifacts:
    paths:
      - build/image

The order of operations and things to special note is:

• Since I'm also dealing with Rootless buildah/stable image not working #3053 (comment) this is running on Gitlab's SaaS runners which are docker executors.
• In order to have qemu-user-static I'm installing podman and using a remote docker sock for it install the binary fmt bits
• Buildah in CI does some arbitrary things but it's expected of the user to build the container in a script (will show an example below)
• Since a parallel matrix is used, the expected usage is that amd64 and arm64 builds simultaneously run pushing their respective tags to the registry.

an example build script. which get's executed in GitLab CI and $ARCH gets populated

#!/usr/bin/env bash
set -x

ctr1=$(buildah from --arch "$ARCH" registry.fedoraproject.org/fedora)

buildah run "$ctr1" -- echo hi

## Include some buildtime annotationssa
buildah config --annotation "com.example.build.host=$(uname -n)" "$ctr1"

## Run our server and expose the port
buildah config --cmd "/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf" "$ctr1"
buildah config --port 80 "$ctr1"

This results in each arch (arm64, amd64) getting built successfully though failing to push arm64.

error pushing image "registry.gitlab.com/kutara/container-images/test:test-5855f0bc" to "docker://registry.gitlab.com/kutara/container-images/test:test-5855f0bc": Error choosing an image from manifest list containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@6094565d6e25fc5c5b59e2792d8f32907b1c1b9cfd2d1496409d228a11c9d31c: no image found in image index for architecture amd64, variant "", OS linux

This somewhat makes sense.. since this job only built for arm64 and doesn't have a amd64 image but how do I instruct buildah to not worry about amd64 when pushing?

[anthr76]

Curiously:

Simplifying the process with something like:

.buildah native:
  image: quay.io/buildah/stable:v1.21.0
  extends:
    - .container build
  services:
    - docker:dind
  variables:
    FF_ENABLE_BASH_EXIT_CODE_CHECK: "true"
  before_script:
    - mkdir -p build
    - export CONTAINER_HOST=tcp://docker:2375/
    - echo "Building ${GL_BUILD_TAG} ${QUAY_BUILD_TAG} using ${SCRIPT_DIR}/${SCRIPT_NAME}"
    - mkdir -p /run/containers/$UID/
    - echo "$DOCKERCFG" > /run/containers/$UID/auth.json
    - export fed_ver=$(echo $FGC | tr -d f)
    - mkdir -p /var/cache/dnf/$fed_ver
    - dnf install -y podman
    - sudo podman run --rm --privileged docker.io/multiarch/qemu-user-static --reset -p yes
  script:
    - bash -e $SCRIPT_DIR/$SCRIPT_NAME
    - export CONTAINER=$(buildah containers --format "{{.ContainerName}}")
    - buildah commit $CONTAINER $GL_BUILD_TAG
    - buildah commit $CONTAINER $QUAY_BUILD_TAG
    - buildah push -f oci --tls-verify=false $GL_BUILD_TAG
    - buildah push $QUAY_BUILD_TAG
    - buildah push -f docker $GL_BUILD_TAG oci:build/image
  parallel:
    matrix:
      - ARCH:
        - amd64
        - arm64
  cache:
    paths:
      - /var/cache/dnf
  artifacts:
    paths:
      - build/image

Results in whatever build finishes first wins first for the arch.... /shrug

  "Architecture": "arm64",
  "Os": "linux"