Can not pull (some) images after systemd upgrade that enabled cgroupsv2 by default

[trusch]

I have two laptops and I kept one of them using cgroups v1 to be able to still use docker for some work related tasks (mainly docker-compose). My other laptop is cgroupsv2 and podman only. Both of them are running archlinux and are updated regularly.

Now after one of the last updates systemd got updated on my work laptop and this resulted in cgroupsv2 being mounted by default. This shouldn't be a problem, since docker now also supports this and therefore there is no reason to stick on v1.

The problem is that podman just stopped working properly on that laptop while its running fine on my other laptop (which is on the same kernel/systemd/podman versions, also the same root filesystem which is ext4) I really can't spot any differences.

The problem occurs when I pull a bigger image like ubuntu, golang or rust images. It works fine with alpine or debian though.

(beast) ➜ /etc podman pull ubuntu
✔ docker.io/library/ubuntu:latest
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob a70d879fa598 skipped: already exists  
Copying blob c4394a92d1f8 skipped: already exists  
Copying blob 10e6159c56c0 done  
Copying config 26b77e5843 done  
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): operation not permitted
Error: Error committing the finished image: error adding layer with blob "sha256:10e6159c56c084c858f5de2416454ac0a49ddda47b764e4379c5d5a147c9bf5f": Error processing tar file(exit status 1): operation not permitted

I searched for the error and found some older issues that hinted me at using cgroupfs as cgroup manager, but this also doesn't help. Increasing the log-level also doesn't show any meaningfull hints. I cleaned my ~/.local/share/containers completely, reinstalled podman, tried using the latest master version (yesterday), using runc instead of crun, but all to no success.

Here is the podman infoof the broken machine, I really hope someone might have another idea. I'd really like to avoid just freshly setting up the laptop.

host:
  arch: amd64
  buildahVersion: 1.20.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon ist in conmon 1:2.0.27-1 enthalten
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 65fad4bfcb250df0435ea668017e643e7f462155'
  cpus: 8
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: beast
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.11-arch1-1
  linkmode: dynamic
  memFree: 29159882752
  memTotal: 33387319296
  ociRuntime:
    name: crun
    package: /usr/bin/crun ist in crun 0.19-1 enthalten
    path: /usr/bin/crun
    version: |-
      crun version 0.19
      commit: e67a75672412975916dac6b87f8346f870e4b99a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns ist in slirp4netns 1.1.9-1 enthalten
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 0
  swapTotal: 0
  uptime: 24h 29m 37.72s (Approximately 1.00 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/tino/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tino/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /home/tino/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.0
  Built: 1617202008
  BuiltTime: Wed Mar 31 16:46:48 2021
  GitCommit: 9f09fb62cba8f44c18eda84db3e72aab3ae76046-dirty
  GoVersion: go1.16.2
  OsArch: linux/amd64
  Version: 3.1.0

Pulling as root works like expected...

I would really appreciate any hints, thanks in advance!

[trusch]

I think this is just this issue here: #9936
But I still don't get why its working on my second laptop which has the very same configuration.