Buildah fails with Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied

[codethief]

I'm trying to run Buildah inside a Gitlab job running as rootful container in an Azure-managed k8s cluster (AKS) with containerd. According to the Gitlab docs this should be possible as long as I choose vfs as storage driver.

However, both buildah info and

buildah build \
  --isolation chroot \
  --cache-from "$IMAGE_CACHE_URL" \
  --cache-to "$IMAGE_CACHE_URL" \
  --file "$DOCKERFILE" \
  --format docker \
  --layers \
  --retry 5 \
  --secret=id=envrc,src=./.envrc \
  --storage-driver vfs \
  --tag "$IMAGE_URL" \
  "$DOCKER_CONTEXT"

immediately fail with

Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied
time="2024-09-13T23:27:46Z" level=warning msg="failed to shutdown storage: \"mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied\""

Now, I should mention our containers run with AppArmor enabled,

$ cat /proc/$$/attr/current
cri-containerd.apparmor.d (enforce)

and after a bit of research I found the very similar-sounding #2899 , which pointed me to https://github.com/containerd/containerd/blob/main/contrib/apparmor/template.go (= the cri-containerd.apparmor.d profile) which indeed contains a line deny mount, so I suspect this is the issue here. (Unfortunately, I have no way to truly verify this, because I don't have access to our k8s / containerd settings.)

In any case, my question is: What does buildah build --storage-driver=vfs --isolation=chroot need mount permissions for in the first place? So far my understanding had been that that vfs does not involve any mounting(?) Also, why does buildah info need to do any mounting?

On a general note, given that running Buildah nested inside another container is such a common use case, I think it'd be great to have an overview in the docs outlining which buildah settings require what capabilities & permissions. Trying to understand what's possible when it comes to container nesting and what isn't has been a rather painful trial & error experience.

[rhatdan]

Mounting /proc and /sys inside of the user namespace for the chroot, I would guess.

[nalind]

Changing mount propagation (give or take the storage "skip_mount_home" option), handling bind mounts (/proc and /sys, as @rhatdan noted, plus most of the contents of /dev since we can't mknod(2)), any RUN --mount flags, the --build-context and -v flags for buildah build... it's not something we'd want to try working around.

[codethief]

Thanks to both of you! I guess the conclusion is that running containers is not really possible without mount permissions. That's unfortunate but I've learned something new at least. :)