Aurora-dx-nvidia No Longer Automatically Applying Updates After Reboot

[Mortifier13]

Describe the bug

I have 2 systems (desktop and laptop) running aurora-dx-nvidia on different branches - latest and stable. I’ve noticed in the last few weeks ((around the 7th or so), the updates are downloading but they are not applying after reboot. I have to manually rollback to a previous version (the 0801 one) then do a rpm-ostree update in order for updates to apply on reboot. I had to do that with the laptop on stable last week and it just downloaded a new update for this week and will not apply it automatically. Here is the rpm-ostree status after rebooting when it did not apply:

What did you expect to happen?

❯ rpm-ostree status -v
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx-nvidia:stable (index: 0)
Digest: sha256:3c79afa3f5dbb3f8db4e0a4b243b80db31b5b689b63e781094e31dc20000fb5e
Version: 40.20240813.0 (2024-08-13T06:35:52Z)
Commit: 48c1d27ab8d7b8cb967f1e4aa906abf1b2eb65adf672ee5825f69bfd6a251154
Staged: no
StateRoot: fedora

ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx-nvidia:stable (index: 1)
Digest: sha256:392c885695d7b8e4c89c42ee21e94c0f8f86647ffdb343dd2c99c21ae397432f
Version: 40.20240801.0 (2024-08-01T19:56:00Z)
Commit: 2ad3671f2be2b081e4048533e18c120682684f5b6a7e7145b64fe80bc0051dea
StateRoot: fedora

AvailableUpdate:
Timestamp: 2024-08-20T05:52:32Z
Commit: 6d82f58c00667266c94380faa7ff7f75ca2b134330dcb8d7246478a6f4d8f782
Upgraded: amd-gpu-firmware 20240709-1.fc40 → 20240811-2.fc40
amd-ucode-firmware 20240709-1.fc40 → 20240811-2.fc40
atheros-firmware 20240709-1.fc40 → 20240811-2.fc40
brcmfmac-firmware 20240709-1.fc40 → 20240811-2.fc40
breeze-icon-theme 6.5.0-1.fc40 → 6.5.0-2.fc40
btrfs-progs 6.9.2-1.fc40 → 6.10-1.fc40
buildah 1.36.0-1.fc40 → 1.37.0-1.fc40
cirrus-audio-firmware 20240709-1.fc40 → 20240811-2.fc40
cockpit-machines 316-1.fc40 → 317-1.fc40
cockpit-ostree 1:203-1.fc40 → 1:204-1.fc40
cockpit-podman 91-1.fc40 → 92-1.fc40
code 1.92.1-1723066391.el8 → 1.92.2-1723661076.el8
devpod v0.5.18-1.fc40 → v0.5.19-1.fc40
distribution-gpg-keys 1.104-1.fc40 → 1.105-1.fc40
docker-buildx-plugin 0.16.1-1.fc40 → 0.16.2-1.fc40
docker-ce 3:27.1.1-1.fc40 → 3:27.1.2-1.fc40
docker-ce-cli 1:27.1.1-1.fc40 → 1:27.1.2-1.fc40
docker-ce-rootless-extras 27.1.1-1.fc40 → 27.1.2-1.fc40
egl-gbm 2:1.1.1-5.20240412git649c079.fc40 → 2:1.1.2-1.fc40
egl-gbm 2:1.1.1-5.20240412git649c079.fc40 → 2:1.1.2-1.fc40
egl-wayland 1.1.15-1.fc40 → 1.1.15-2.20240814gitf30cb0e.fc40
egl-wayland 1.1.15-1.fc40 → 1.1.15-2.20240814gitf30cb0e.fc40
ethtool 2:6.9-1.fc40 → 2:6.10-1.fc40
fzf 0.54.1-1.fc40 → 0.54.3-1.fc40
gperftools-libs 2.14-3.fc40 → 2.14-4.fc40
ibus 1.5.30-5.fc40 → 1.5.30-6.fc40
ibus-gtk2 1.5.30-5.fc40 → 1.5.30-6.fc40
ibus-gtk3 1.5.30-5.fc40 → 1.5.30-6.fc40
ibus-gtk4 1.5.30-5.fc40 → 1.5.30-6.fc40
ibus-libs 1.5.30-5.fc40 → 1.5.30-6.fc40
ibus-panel 1.5.30-5.fc40 → 1.5.30-6.fc40
ibus-setup 1.5.30-5.fc40 → 1.5.30-6.fc40
intel-audio-firmware 20240709-1.fc40 → 20240811-2.fc40
intel-gpu-firmware 20240709-1.fc40 → 20240811-2.fc40
intel-media-driver 24.1.5-1.fc40 → 24.2.5-1.fc40
iwlegacy-firmware 20240709-1.fc40 → 20240811-2.fc40
iwlwifi-dvm-firmware 20240709-1.fc40 → 20240811-2.fc40
iwlwifi-mvm-firmware 20240709-1.fc40 → 20240811-2.fc40
kcli 99.0.0.git.202408112231.ec8c845-0.fc40 → 99.0.0.git.202408152041.5eeaa03-0.fc40
kde-settings 40.0-1.fc40 → 40.1-1.fc40
kde-settings-plasma 40.0-1.fc40 → 40.1-1.fc40
kde-settings-pulseaudio 40.0-1.fc40 → 40.1-1.fc40
kde-settings-sddm 40.0-1.fc40 → 40.1-1.fc40
kernel 6.9.7-200.fc40 → 6.9.11-200.fc40
kernel-core 6.9.7-200.fc40 → 6.9.11-200.fc40
kernel-modules 6.9.7-200.fc40 → 6.9.11-200.fc40
kernel-modules-core 6.9.7-200.fc40 → 6.9.11-200.fc40
kernel-modules-extra 6.9.7-200.fc40 → 6.9.11-200.fc40
kernel-tools 6.10.3-200.fc40 → 6.10.5-200.fc40
kernel-tools-libs 6.10.3-200.fc40 → 6.10.5-200.fc40
kf6-breeze-icons 6.5.0-1.fc40 → 6.5.0-2.fc40
libedit 3.1-51.20240517cvs.fc40 → 3.1-53.20240808cvs.fc40
libedit 3.1-51.20240517cvs.fc40 → 3.1-53.20240808cvs.fc40
libedit-devel 3.1-51.20240517cvs.fc40 → 3.1-53.20240808cvs.fc40
libertas-firmware 20240709-1.fc40 → 20240811-2.fc40
libimagequant 4.0.3-3.fc40 → 4.0.3-5.fc40
libldb 2.9.1-1.fc40 → 2.9.1-4.fc40
libnfsidmap 1:2.6.4-0.rc6.fc40 → 1:2.6.4-0.rc8.fc40
libppd 1:2.0.0-4.fc40 → 1:2.0.0-6.fc40
libsrtp 2.3.0-14.fc40 → 2.6.0-1.fc40
linux-firmware 20240709-1.fc40 → 20240811-2.fc40
linux-firmware-whence 20240709-1.fc40 → 20240811-2.fc40
mt7xxx-firmware 20240709-1.fc40 → 20240811-2.fc40
nfs-utils 1:2.6.4-0.rc6.fc40 → 1:2.6.4-0.rc8.fc40
nvidia-gpu-firmware 20240709-1.fc40 → 20240811-2.fc40
nxpwireless-firmware 20240709-1.fc40 → 20240811-2.fc40
openssl 1:3.2.1-2.fc40 → 1:3.2.2-3.fc40
openssl-libs 1:3.2.1-2.fc40 → 1:3.2.2-3.fc40
passt 0^20240726.g57a21d2-1.fc40 → 0^20240814.g61c0b0d-1.fc40
passt-selinux 0^20240726.g57a21d2-1.fc40 → 0^20240814.g61c0b0d-1.fc40
podman 5:5.1.2-1.fc40 → 5:5.2.0-1.fc40
podmansh 5:5.1.2-1.fc40 → 5:5.2.0-1.fc40
python3-boto3 1.34.153-1.fc40 → 1.34.162-1.fc40
python3-botocore 1.34.153-1.fc40 → 1.34.162-1.fc40
qt-settings 40.0-1.fc40 → 40.1-1.fc40
realtek-firmware 20240709-1.fc40 → 20240811-2.fc40
rpm-ostree 2024.6-1.fc40 → 2024.7-1.fc40
rpm-ostree-libs 2024.6-1.fc40 → 2024.7-1.fc40
skopeo 1:1.15.2-1.fc40 → 1:1.16.0-1.fc40
tailscale 1.70.0-1.fc40 → 1.72.0-1
tiwilink-firmware 20240709-1.fc40 → 20240811-2.fc40
vim-common 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40
vim-data 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40
vim-enhanced 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40
vim-filesystem 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40
vim-minimal 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40
wpa_supplicant 1:2.11-2.fc40 → 1:2.11-3.fc40
xxd 2:9.1.660-1.fc40 → 2:9.1.672-1.fc40
Removed: displaylink-6.0.0-2.fc40.x86_64
gvisor-tap-vsock-6:0.7.4-1.fc40.x86_64
gvisor-tap-vsock-gvforwarder-6:0.7.4-1.fc40.x86_64
kmod-evdi-6.9.7-200.fc40.x86_64-1.14.5-2.20240726giteab561a.fc40.x86_64
kmod-kvmfr-6.9.7-200.fc40.x86_64-0.0.git.23.2de42028-1.fc40.x86_64
kmod-nvidia-6.9.7-200.fc40.x86_64-3:560.31.02-1.fc40.x86_64
kmod-openrazer-6.9.7-200.fc40.x86_64-100.0.0.git.530.886f986d-1.fc40.x86_64
kmod-v4l2loopback-6.9.7-200.fc40.x86_64-0.13.1-1.fc40.x86_64
kmod-wl-6.9.7-200.fc40.x86_64-6.30.223.271-51.fc40.x86_64
kmod-xone-6.9.7-200.fc40.x86_64-0.0.git.115.fdbb71f1-1.fc40.x86_64
kmod-xpadneo-6.9.7-200.fc40.x86_64-0.9.6-2.20240423git73be2eb.fc40.x86_64
kmod-zfs-6.9.7-200.fc40.x86_64-2.2.5-1.fc40.x86_64
libevdi-1.14.5-2.20230726giteab561a.fc40.x86_64
Added: kmod-kvmfr-6.9.11-200.fc40.x86_64-0.0.git.23.2de42028-1.fc40.x86_64
kmod-nvidia-6.9.11-200.fc40.x86_64-3:560.31.02-1.fc40.x86_64
kmod-openrazer-6.9.11-200.fc40.x86_64-100.0.0.git.530.886f986d-1.fc40.x86_64
kmod-v4l2loopback-6.9.11-200.fc40.x86_64-0.13.1-1.fc40.x86_64
kmod-wl-6.9.11-200.fc40.x86_64-6.30.223.271-51.fc40.x86_64
kmod-xone-6.9.11-200.fc40.x86_64-0.0.git.115.fdbb71f1-1.fc40.x86_64
kmod-xpadneo-6.9.11-200.fc40.x86_64-0.9.6-2.20240423git73be2eb.fc40.x86_64
kmod-zfs-6.9.11-200.fc40.x86_64-2.2.5-1.fc40.x86_64

~
❯

Output of rpm-ostree status

No response

Output of groups

No response

Extra information or context

No response

[lion7]

My automatic updates on Aurora stopped working as well, with the following error:

❯ rpm-ostree upgrade 
note: automatic updates (stage) are enabled
Pulling manifest: ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: cryptographic signature verification failed: invalid signature when validating ASN.1 encoded signature

I think this is due to ublue-os/main#599 and #1483.
As a workaround I fixed it by accepting all signatures in /etc/containers/policy.json:

/etc/containers🔒 
❯ diff policy.json.bak policy.json -C 2
*** policy.json.bak	2024-08-22 09:46:37.796002073 +0200
--- policy.json	2024-08-22 09:46:53.903880061 +0200
***************
*** 32,40 ****
              "ghcr.io/ublue-os": [
                  {
!                     "type": "sigstoreSigned",
!                     "keyPath": "/usr/etc/pki/containers/ublue-os.pub",
!                     "signedIdentity": {
!                         "type": "matchRepository"
!                     }
                  }
              ],
--- 32,36 ----
              "ghcr.io/ublue-os": [
                  {
!                     "type": "insecureAcceptAnything"
                  }
              ],

[Mortifier13]

Sounds like a different issue from the one I'm having. The issue I'm having is occurring on 2 independent systems installed separately and there's no errors when rpm-ostree update is run.

[m2Giles]

@lion7 your error is indeed the one that you linked. See the pinned issue to get the updated cosign public key instead of doing insecureAcceptAnything.

@Mortifier13 is this still occuring? It sounds like the deployment is failing; however, its confusing since you said an update works after a rollback. The only time I've had a similar issue was when I had specifically modified the root fs outside of an ostree update and then removed a package dependent on those changes.

[Mortifier13]

@lion7 your error is indeed the one that you linked. See the pinned issue to get the updated cosign public key instead of doing insecureAcceptAnything.

@Mortifier13 is this still occuring? It sounds like the deployment is failing; however, its confusing since you said an update works after a rollback. The only time I've had a similar issue was when I had specifically modified the root fs outside of an ostree update and then removed a package dependent on those changes.

I think my issue is occurring due to a lack of free space on /root, it was almost full when I had a few pinned deployments so I removed some of them so I'm down to 3 total deployments in /boot/ostree/. My /boot is 1GB and with only 3 now instead of 5 it's still using about 750 megs. It worked on a manual update last night so I'm testing it over the next few days to see but there haven't been many daily updates rolling through the nvidia branch lately. If it seems to work on updating I'd like to increase the size of my /boot partition since I like to always keep a few pinned deployments on the latest branch.