Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman cannot run [conmon:e] Include journald in compilation path to log to systemd journal #348

Open
b3n3w opened this issue Jul 29, 2022 · 9 comments · May be fixed by #431
Open

podman cannot run [conmon:e] Include journald in compilation path to log to systemd journal #348

b3n3w opened this issue Jul 29, 2022 · 9 comments · May be fixed by #431

Comments

@b3n3w
Copy link

b3n3w commented Jul 29, 2022

Hi there,

on one of our machines were not able to run any podman container due to the following error:

[conmon:e] Include journald in compilation path to log to systemd journal
Error: write child: broken pipe

I already reinstalled podman, conmon and checked the permissions of /var/log/journal.
When running as sudo the operation works without problem.

The user is already in the systemd-journal group

Debug output of podman:
DEBU[0000] mounted container "fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0" at "/home/user/.local/share/containers/storage/overlay/4d5225f292fbc66dff495c31621fcb05193e294a5b81d9188687d441ff88a140/merged" DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-f4ef9681-4344-7949-fa74-162c74242d8f tap0 DEBU[0000] Created root filesystem for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 at /home/user/.local/share/containers/storage/overlay/4d5225f292fbc66dff495c31621fcb05193e294a5b81d9188687d441ff88a140/merged DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription DEBU[0000] Setting CGroups for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 to user.slice:libpod:fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0000] Workdir "/" resolved to host path "/home/user/.local/share/containers/storage/overlay/4d5225f292fbc66dff495c31621fcb05193e294a5b81d9188687d441ff88a140/merged" DEBU[0000] Created OCI spec for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 at /home/user/.local/share/containers/storage/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata/config.json DEBU[0000] /usr/local/libexec/podman/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/local/libexec/podman/conmon args="[--api-version 1 -c fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 -u fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 -r /usr/bin/crun -b /home/user/.local/share/containers/storage/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata -p /run/user/1000/containers/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata/pidfile -n admiring_dewdney --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l journald --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/user/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0]" INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0.scope [conmon:e] Include journald in compilation path to log to systemd journal DEBU[0000] Cleaning up container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 DEBU[0000] Tearing down network namespace at /run/user/1000/netns/cni-f4ef9681-4344-7949-fa74-162c74242d8f for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 DEBU[0000] unmounted container "fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0" DEBU[0000] ExitCode msg: "write child: broken pipe" Error: write child: broken pipe

@jklaiho
Copy link

jklaiho commented Aug 17, 2022

I'm seeing the same thing. While deploying a Ubuntu 22.04 server, I downloaded https://github.com/containers/conmon/releases/download/v2.1.3/conmon.amd64, and am unable to use log_driver="journald" in containers.conf with it.

Seems related to containers/podman#5467

@haircommander
Copy link
Collaborator

@saschagrunert it seems our static builds are missing a journald library can you PTAL

@saschagrunert
Copy link
Member

I guess we did exclude systemd (libsystemd-journald and libsystemd) on purpose because it includes a huge amount of dependencies. Also statically building systemd incorporated some troubles in the past. I see that crun does statically link systemd via nix, so we may wanna copy over the bits from there.

@b3n3w
Copy link
Author

b3n3w commented Aug 22, 2022

I guess we did exclude systemd (libsystemd-journald and libsystemd) on purpose because it includes a huge amount of dependencies. Also statically building systemd incorporated some troubles in the past. I see that crun does statically link systemd via nix, so we may wanna copy over the bits from there.

Do you have any recommendations how to get conmon and podman running again ? I already tried to uninstall every related service and reinstall them.

@mullerch
Copy link

mullerch commented Sep 2, 2022

I had the same issue. My workaround is to drop logs with podman run --log-driver none.

@jklaiho
Copy link

jklaiho commented Mar 14, 2023

With no recent developments on this issue, I thought I'd share what I did to install conmon from source in a way that supports journald logging on Ubuntu 22.04:

apt install -y libglib2.0-dev libseccomp-dev libsystemd-dev
<download and extract https://github.com/containers/conmon/archive/refs/tags/v<VERSION_HERE>.tar.gz somewhere like /tmp, then in the extracted directory:>
GOCACHE="$(mktemp -d)" make PREFIX=/usr
GOCACHE="$(mktemp -d)" make PREFIX=/usr podman

After this, /usr/libexec/podman/conmon exists. Podman itself doesn't need to be installed at this stage yet. Note: PREFIX=/usr might not be strictly necessary, I think it defaults to /usr/local which may be fine for you, but I recall having some issues down the line when installing/configuring other Podman dependencies without it. I use the same prefix later when building Podman itself from source. YMMV.

@haircommander
Copy link
Collaborator

I have attempted to add systemd to the nix builds in #431

It looks like the upstream debian packages add systemd libraries for the builds, so we should be good there. Are folks out there still using OBS builds/does support need to be added there?

@stellarpower
Copy link

I am using the Kubic PPA and this suddenly surfaced for me. Interestingly, I didn't do anything, so guess unattended-upgrades maybe kicked in. If it's possible to add to those builds too, that would be appreciated; otherwise will just turn logging off for the time being as a workaround.

@AdamWyzgol
Copy link

Hi, any news in that issue? I see PR that could fix, anyone is working on that?

hswong3i added a commit to alvistack/containers-conmon that referenced this issue Jun 17, 2024
    git clean -xdf
    tar zcvf ../conmon_2.1.10.orig.tar.gz --exclude=.git .
    debuild -uc -us
    cp conmon.spec ../conmon_2.1.10-1.spec
    cp ../conmon*2.1.10*.{gz,xz,spec,dsc} /osc/home\:alvistack/containers-conmon-2.1.10/
    rm -rf ../conmon*2.1.10*.*

See containers#348

Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
hswong3i added a commit to alvistack/containers-conmon that referenced this issue Jun 17, 2024
    git clean -xdf
    tar zcvf ../conmon_2.1.11.orig.tar.gz --exclude=.git .
    debuild -uc -us
    cp conmon.spec ../conmon_2.1.11-1.spec
    cp ../conmon*2.1.11*.{gz,xz,spec,dsc} /osc/home\:alvistack/containers-conmon-2.1.11/
    rm -rf ../conmon*2.1.11*.*

See containers#348

Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
hswong3i added a commit to alvistack/containers-conmon that referenced this issue Jun 17, 2024
    git clean -xdf
    tar zcvf ../conmon_2.1.12.orig.tar.gz --exclude=.git .
    debuild -uc -us
    cp conmon.spec ../conmon_2.1.12-1.spec
    cp ../conmon*2.1.12*.{gz,xz,spec,dsc} /osc/home\:alvistack/containers-conmon-2.1.12/
    rm -rf ../conmon*2.1.12*.*

See containers#348

Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants