podman ps freeze / ver 5.1.2 -> 5.2.1 fedora

[kraftreich]

Since the update to 5.2.1 I can't use some podman commands like
podman ps or podman auto-update
When I exec one of these, it simply does nothing, just sleeping, need to press Strg+C to get back my terminal.
The strange thing is

• it worked for 5.1.2
• all my pods do their job, so they are working. But on terminal I can't eg update them.
• the problem is only for some pods. (I have for every pod a seperated user. So I have some users who can do podman auto-update and some who can't)

I didn't find some common (miss)configuration for the freezing podman ps pods.
Maybe you have some idea where to look to get things right.
Thx
Chris
Fedora 40 / AMD64/ latest updates 23.8.24 / podman 5.2.1 / containers with quadlet in a pod per user

[Luap99]

Do you have another podman process around during the hang? This sounds like a deadlock where another process deadlocked and holds the container/pod/volume lock forever.

run kill -ABRT <pid> on the podman process when it hangs and provide the output this will shows us where it hangs.

[kraftreich]

Thx,
I got to disable one by one every pod and check when the problem disappears.
Since the system and all pods are doing the job, I thought the pods are not the problem.
I am using podman auto-update a lot, so I am sure, the problem started the last days.
Is it possible, that a container from one user, locks the container of another user? (different host uid) This shouldnt be possible with rootless users, or?
Then I have 3 problems, cause 3 users have this locking issues.
I will find out with trial and error.
Thx a lot..

[Luap99]

Is it possible, that a container from one user, locks the container of another user? (different host uid) This shouldnt be possible with rootless users, or?

All users are fully separate so it should be impossible that one user caused a deadlock for another.

Just check your user processes with something like ps xww and see what other podman processes are running in parallel. Then you have to figure out which of them is holding the lock.

podman system locks should show which lock id is being held although it doesn't show which container/pod/volume uses that id so it is likely not really useful to you

[kraftreich]

Your infos helped a lot.
This is the problem:
Something blocks
podman[223807]: 2024-08-24 12:07:21.612283733 +0200 CEST m=+0.019836290 image pull 2f1b7c91f38d06214a87fa007d2e1e412a2dc4d455770e0d1b9be17ae77152f2 docker.io/goacme/lego:latest`
after timeout it is terminated by systemd and aborting
Locks don't get removed
Higher timeout wont help, disk has plenty of space. All other containers are running and doing well.
Because of the lock not getting removed, podman xxx commands dont work.

I did a podman system reset, and now also the podman locking problem is gone, and the image gets pulled.

• host network connection can't be the problem
• podman run configuration can't be the problem. I am using quadlets and didn't change a single line before and after podman system reset
• did everything after a fresh reboot, so no outdated stuff from fedora update

I have 2 users on the host left with the same problem, I know now how to get them back, but I think there was a change from 5.1 -> 5.2 that breaks something, podman system reset cleans it up. I can try something different, for the remaining two, if it will help to debug the problem. If not needed, I will reset them....
Thx for the help!
chris
my system is pretty clean, standard fedora server image v40 only used for podman and all I added is /etc/containers/containers.conf
[network] pasta_options = ["-a", "10.0.2.0", "-n", "24", "-g", "10.0.2.2", "--dns-forward", "10.0.2.3", "-a", "fd00::1", "-g", "fd00::2", "-m", "1500","--no-dhcpv6","--no-dhcp"]

[Luap99]

podman[223807]: 2024-08-24 12:07:21.612283733 +0200 CEST m=+0.019836290 image pull 2f1b7c91f38d06214a87fa007d2e1e412a2dc4d455770e0d1b9be17ae77152f2 docker.io/goacme/lego:latest`
after timeout it is terminated by systemd and aborting
Locks don't get removed

This is not how locks work. If there process dies the lock is freed automatically. So you must check for other running podman processes that could hold the lock.
The image pull part isn't hanging, the podman ps stack strace shows it waiting for the container lock so there must be another process holding that lock. You never showed how you run these pods/containers so all I can do is guessing.

If this reproduces easily than this is certainly a big problem.

[kraftreich]

This is the setup working now with podman system reset
nginx+sftpgo container are running in the pod, legoWWW container outside the pod.
All running with same uid, nothing else running at this uid
once all setup
loginctl enable-linger username
systemctl --user daemon-reload
systemctl --user restart nginx-pod aso....

vi .config/containers/systemd/nginx.pod

# vi: ft=systemd
[Pod]
PodName=nginx
PublishPort=9080:8080
PublishPort=9443:8443
PublishPort=9022:2022

vi .config/containers/systemd/sftpgo.container
https://pastebin.com/cvdFxV9K

vi .config/containers/systemd/nginx.container
https://pastebin.com/siNJksiu

vi .config/containers/systemd/legoWWW.container
https://pastebin.com/F6JCSLr5

[kraftreich]

podman system reset didn't help in any way, after reboot podman was locked again.......
So I did some more trial and error.
The container, which locks podman is legoWWW.container https://pastebin.com/F6JCSLr5
If I start it right after boot with WantedBy=multi-user.target default.target, I get the lock problem.
After removing WantedBy, no problems anymore, even when I start the container with systemctl --user restart there is no locking problem.
I thought, this might be like #22197, and tried the workaround #22197 (comment), but didn't help. So I will stay with disabled start on boot.
Interesting:
I had some uids with legoWWW.container running on the machine without the locking problem. So I think it depends on when the legoWWW.containers get started during the boot process, if they lock without releasing, or won't.

[kraftreich]

With podman 5.2.3 this is fixed...
Thx

[Luap99]

Good to know, in that case the issue was most likely #24024