podman build vs podman run

[daipok]

We have a very specific scenario around building images based out from rockylinux, version 9

Just for the current time being, one of the component coming in with yum -y update, have files needing to be extracted to system directory like dev.

We have a container at k8s that we run podman to build images, and then it fails just for this round of yum update (it's been working fine for a long while). We tested different scenarios, and came to this point that, if we build the Dockerfile (USER root), it will fail (just a normal podman build -t testimage:test .)

If we just do

podman run --rm rockylinux:9 yum update -y

then it works

I have a suspicion around user not being elevated (or mapped) properly when the image is being built (when the RUN is run). Have you guys hit this kind of issue before?

[podman@container-builder-8596878565-fg2lb /]$ podman version
Client: Podman Engine
Version: 4.9.4
API Version: 4.9.4
Go Version: go1.21.8
Built: Tue Mar 26 09:39:52 2024
OS/Arch: linux/amd64

[podman@container-builder-8596878565-fg2lb /]$ podman unshare cat /proc/self/uid_map
0 1000 1
1 1 999
1000 1001 64535
65535 100000 564700753

[nalind]

It would be easier to help troubleshoot this if you included the specific error messages and logs produced when something fails, perhaps with the --log-level option being used to increase the amount of information being logged about what's happening when things go awry.

[daipok]

The error is actually when I am doing yum update on the official rockylinux:9, it has to upgrade the filesystem package:

Running scriptlet: setup-2.13.7-10.el9.noarch 6/189
Upgrading : filesystem-3.16-5.el9.x86_64 7/189Error unpacking rpm package filesystem-3.16-5.el9.x86_64

Upgrading : basesystem-11-13.el9.0.1.noarch 8/189
error: unpacking of archive failed on file /dev: cpio: chown failed - Inappropriate ioctl for device
error: filesystem-3.16-5.el9.x86_64: install failed

And then we have two thoughts:

1. This has something to do with the way we run rootless/unprivileged container.
2. But puzzles me I can do the same with the podman run rockylinux:9 yum -y update without any issues

I have some suspicion that it has to extract files to /dev, and how the mounted filepath got into this issue

I tried loglevel=debug and it spits out quite a bit of info

[nalind]

Reporting that it got an ENOTTY error back from a chown() attempt sure sounds weird. Things to compare between the two scenarios would include the type of /dev (stat /dev and stat -f /dev) and the capabilities that the shell running yum has (capsh --decode=$(grep CapEff /proc/self/status | cut -f2)).

Then again, the chroot isolation bits which tend to be used if podman build is running inside of a container are going to create a read-only bind-mount from the container's /dev into the space where the RUN command will be, and attempting to change the ownership of that (if it doesn't match what's in the package) would be expected to produce an error, but that's not one I'd expect to get back in that scenario.

[daipok]

interesting, trying to compare the two:

podman run
stat
File: /dev Size: 340 Blocks: 0 IO Block: 4096 directory Device: 20000eh/2097166d Inode: 1 Links: 5 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-01-24 01:21:01.193712970 +0000 Modify: 2025-01-24 01:21:01.194712961 +0000 Change: 2025-01-24 01:21:01.194712961 +0000 Birth: 2025-01-24 01:21:01.193712970 +0000
stat -f
File: "/dev" ID: 9e0d5b30755818ae Namelen: 255 Type: tmpfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 16384 Free: 16384 Available: 16384 Inodes: Total: 4061427 Free: 4061411
capsh --decode=$(grep CapEff /proc/self/status | cut -f2)
0x0000000000000000=

podman build
stat
File: /dev Size: 380 Blocks: 0 IO Block: 4096 directory Device: 200006h/2097158d Inode: 1 Links: 5 Access: (0755/drwxr-xr-x) Uid: (65534/ nobody) Gid: (65534/ nobody) Access: 2025-01-20 19:35:39.049453189 +0000 Modify: 2025-01-20 19:35:39.053453152 +0000 Change: 2025-01-20 19:35:39.053453152 +0000 Birth: 2025-01-20 19:35:39.049453189 +0000
stat -f
File: "/dev" ID: 15ce7dd72cab4da5 Namelen: 255 Type: tmpfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 16384 Free: 16384 Available: 16384 Inodes: Total: 4061427 Free: 4061409
capsh --decode=$(grep CapEff /proc/self/status | cut -f2)
0x00000000800405fb=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap

so the /dev being owned by "nobody" seems to be the culprit?