Pi5 failed to contact RP1 firmware #6642

llllllllqq · 2025-02-01T07:16:13Z

Describe the bug

dmesg says:

...
rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware [ 3.957885] rp1-pio: probe of 1f00178000.pio failed with error -2
...

Although nothing strange occurred.
The OS still works fine without reboot.

Steps to reproduce the behaviour

It randomly occurs after sudo reboot --halt and then manually press the on-board power button to power on (with default eeprom config).

~~It also randomly occurs when use raspi-config to change power off behavior to "full power off" and then reboot/poweroff.~~ (Sorry I cannot remember whether it is reboot or poweroff)

Update: It also occurs after a suddenly plug-out the usb-c power cable and then plug-in back.
~~It seems only poweroff-related behaviors will trigger this bug. I tested 30 times reboot, no bugs occurred.~~
Update2: reboot can indeed also trigger this bug, only less frequent.

Device (s)

Raspberry Pi 5

System

Raspberry Pi reference 2024-11-19 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 891df1e21ed2b6099a2e6a13e26c91dea44b34d4, stage4

2025/01/22 00:16:51
Copyright (c) 2012 Broadcom version a7753063 (release) (embedded)

Linux rpi5 6.6.74+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.74-1+rpt1 (2025-01-27) aarch64 GNU/Linux

Logs

1.txt

Additional context

Can it be safely ignored?

The text was updated successfully, but these errors were encountered:

pelwell · 2025-02-03T10:13:13Z

That's interesting - it looks like a dependency/ordering issue, and not one I've seen before:

[    3.922665] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    3.922671] rp1-pio: probe of 1f00178000.pio failed with error -2
...
[    3.932284] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55

I get:

[   13.445656] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[   13.550428] rp1-pio 1f00178000.pio: Created instance as pio0

The code for communicating with the firmware exists in the firmware module, so I thought that would enforce an ordering. Some investigation is required.

llllllllqq · 2025-02-03T10:18:35Z

That's interesting - it looks like a dependency/ordering issue, and not one I've seen before:
[    3.922665] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    3.922671] rp1-pio: probe of 1f00178000.pio failed with error -2
...
[    3.932284] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
I get:
[   13.445656] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[   13.550428] rp1-pio 1f00178000.pio: Created instance as pio0
The code for communicating with the firmware exists in the firmware module, so I thought that would enforce an ordering. Some investigation is required.

Thanks for your reply.
Here is my dmesg when the rp1&pio works fine.
I hope it can help to investigate this issue.

normal.txt

pelwell · 2025-02-03T10:28:07Z

Yes - in the working case the order is reversed:

[    3.955929] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[    3.961971] vc4-drm axi:gpu: bound 107c580000.hvs (ops vc4_hvs_ops [vc4])
[    3.967397] rp1-pio 1f00178000.pio: Created instance as pio0

aBUGSworstnightmare-rpi · 2025-02-03T14:06:31Z

[ 5.557565] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware [ 5.557571] rp1-pio: probe of 1f00178000.pio failed with error -2 [ 5.558534] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55

Seeing that on reboot as well /custom kernel on a Pi5 DUT, trying to get an overlay to work)

Linux Pi5uSDDUT 6.6.74-v8-16k+ #1 SMP PREEMPT Sun Feb 2 10:41:23 CET 2025 aarch64 GNU/Linux Revision : c04170 Serial : cb30eb1fd7e7e672 Model : Raspberry Pi 5 Model B Rev 1.0

pelwell · 2025-02-03T14:12:45Z

It's really strange. I can't get it to fail in that way without hacks - even with a massive delay in the firmware driver, the pio driver waits. Anyway, I'm just testing a fix.

aBUGSworstnightmare-rpi · 2025-02-03T15:03:20Z

seems quite random. Seen it frequently on reboots (running 'sudo reboot' on CLI)

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: raspberrypi#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: raspberrypi#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

mayo228 · 2025-02-03T15:32:58Z

is there a way to fix this? I keep getting this error and I can't boot the OS, should I be worried that I will loose data on my raspberry?

pelwell · 2025-02-03T15:38:10Z

I don't understand why this affecting some people while I can't reproduce it without hacking the driver, but regardless, #6645 should fix the issue. After about 40 minutes you should be able to install a trial kernel with sudo rpi-update pulls/6645, after first backing up any important data.

pelwell · 2025-02-03T15:39:03Z

And no, you shouldn't worry - it would only affect you if you wanted to use piolib to drive the GPIOs.

aBUGSworstnightmare-rpi · 2025-02-03T16:24:00Z

I have no idea if and how below is related but I just wanted to state that I'm not making use of piolib for accessing GPIO's.

[   13.678330] ieee80211 phy0: brcmf_p2p_send_action_frame: Unknown Frame: category 0xa, action 0x8
[   15.374753] Bluetooth: RFCOMM TTY layer initialized
[   15.374765] Bluetooth: RFCOMM socket layer initialized
[   15.374772] Bluetooth: RFCOMM ver 1.11
[   27.660651] brcmfmac: brcmf_set_channel: set chanspec 0xd022 fail, reason -52
[   27.768681] brcmfmac: brcmf_set_channel: set chanspec 0xd026 fail, reason -52
[   27.872656] brcmfmac: brcmf_set_channel: set chanspec 0xd02a fail, reason -52
[   27.976661] brcmfmac: brcmf_set_channel: set chanspec 0xd02e fail, reason -52
[   29.636663] brcmfmac: brcmf_set_channel: set chanspec 0xd090 fail, reason -52
[   29.636875] brcmfmac: brcmf_set_channel: set chanspec 0xd095 fail, reason -52
[   29.637196] brcmfmac: brcmf_set_channel: set chanspec 0xd099 fail, reason -52
[   29.637400] brcmfmac: brcmf_set_channel: set chanspec 0xd09d fail, reason -52
[   29.637594] brcmfmac: brcmf_set_channel: set chanspec 0xd0a1 fail, reason -52
[   29.637791] brcmfmac: brcmf_set_channel: set chanspec 0xd0a5 fail, reason -52

pelwell · 2025-02-03T16:31:54Z

I have no idea if and how below is related

You should know that WiFi and Bluetooth information and warning messages are unrelated.

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: raspberrypi#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

llllllllqq · 2025-02-04T00:15:04Z

It seems after applying sudo rpi-update pulls/6645, this issue disappeared. (I tested about 10 times reboot/halt/suddenly poweroff). Thanks for your work! If anything strange found, I'll post here.

BTW, can I safely stay at this firmware and wait for the next stable release offered by apt? Or do I need a sudo apt install --reinstall raspi-firmware to restore first before the next stable release?

pelwell · 2025-02-04T09:16:11Z

You should be fine with this kernel until the next proper release - I'm not aware of any new issues.

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: raspberrypi#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: raspberrypi#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

randomMesh · 2025-02-07T18:55:28Z

I can confirm the error happens on my PI 5 too:

[    0.692347] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    0.692370] rp1-pio: probe of 1f00178000.pio failed with error -2

I am running the latest firmware:

BOOTLOADER: up to date
   CURRENT: Wed 22 Jan 00:16:51 UTC 2025 (1737505011)
    LATEST: Wed 22 Jan 00:16:51 UTC 2025 (1737505011)
   RELEASE: latest (/usr/lib/firmware/raspberrypi/bootloader-2712/latest)
            Use raspi-config to change the release.

helpful-emu57 · 2025-02-09T03:22:11Z

A week ago I also started receiving these errors during bootup:

rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware

At the same time I also started getting protocol errors for the wireless mouse and wireless keyboard
logitech-hidpp-device - hidpp_root_get_ protocol_version: received protocol error 0x08
(Although - both devices continued to work appropriately.)

Changing to wired devices there are no mouse or keyboard errors but still have the rp1-pio error

I have performed the "sudo rpi-update pulls/6645" and no change was observed.

pelwell · 2025-02-09T08:26:49Z

What does rpi-eeprom-update show?

Support for the RP1 firmware mailbox API is rolling out to Pi 5 EEPROM images. For most users, the fact that the PIO is not available is no cause for alarm. Change the message to a warning, so that it does not appear with "quiet" in cmdline.txt. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

Support for the RP1 firmware mailbox API is rolling out to Pi 5 EEPROM images. For most users, the fact that the PIO is not available is no cause for alarm. Change the message to a warning, so that it does not appear with "quiet" in cmdline.txt. Link: raspberrypi#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

Support for the RP1 firmware mailbox API is rolling out to Pi 5 EEPROM images. For most users, the fact that the PIO is not available is no cause for alarm. Change the message to a warning, so that it does not appear with "quiet" in cmdline.txt. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

ozapft · 2025-03-02T20:57:23Z

I also had the error message “failed to contact rp1 firmware“.
So I installed a new sd card as a test. This error message didn't appear with this new sd card, even though it uses the same kernel. When comparing the /boot/firmware directory, I noticed that the initramfs files on the older card were ~17 MB in size and the two new ones were 12.7 MB. After copying the two files from the new sd card to the old one, the error disappeared.

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

Support for the RP1 firmware mailbox API is rolling out to Pi 5 EEPROM images. For most users, the fact that the PIO is not available is no cause for alarm. Change the message to a warning, so that it does not appear with "quiet" in cmdline.txt. Link: #6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

If the RP1 firmware has reported an error then return that from the PIO probe function, otherwise defer the probing. Link: raspberrypi/linux#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

To avoid pointless retries, let the probe function succeed if the firmware interface is configured correctly but the firmware is incompatible. The value of the private drvdata field holds the outcome. Link: raspberrypi/linux#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

Support for the RP1 firmware mailbox API is rolling out to Pi 5 EEPROM images. For most users, the fact that the PIO is not available is no cause for alarm. Change the message to a warning, so that it does not appear with "quiet" in cmdline.txt. Link: raspberrypi/linux#6642 Signed-off-by: Phil Elwell <phil@raspberrypi.com>

moodeaudio · 2025-03-08T11:59:13Z

I'm also seeing this error sometimes, but only on Pi5 with NVMe (non-boot) drive attached.

RPiOS: 12.9 Bookworm 64-bit | Linux: 6.6.74 64-bit (built with Pi-Gen)
ii  raspi-firmware   1:1.20250305-1  all  Raspberry Pi family GPU firmware and bootloaders

I have 2 such Pi5's and one symptom is that sometimes reboot or power off does not complete and the Pi never reaches the halt state (red LED). Two other Pi5's w/o NVMe drive are fine.

Pi5 with NVMe drive

pi@moode9:~ $ dmesg | grep rp1
[    0.805353] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    0.823933] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    2.284970] rp1 0001:01:00.0: bar0 len 0x4000, start 0x1f00410000, end 0x1f00413fff, flags, 0x40200
[    2.294054] rp1 0001:01:00.0: bar1 len 0x400000, start 0x1f00000000, end 0x1f003fffff, flags, 0x40200
[    2.303317] rp1 0001:01:00.0: enabling device (0000 -> 0002)
[    2.309638] rp1 0001:01:00.0: chip_id 0x20001927
[    2.318624] genirq: irq_chip rp1_irq_chip did not update eff. affinity mask of irq 106
[    5.742889] rp1-firmware: probe of rp1_firmware failed with error -110
[    5.749497] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    5.749505] rp1-pio: probe of 1f00178000.pio failed with error -2
pi@moode9:~ $

Pi5 w/o NVMe drive

pi@moode:~ $ dmesg | grep rp1
[    1.015138] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    1.033523] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    2.137633] rp1 0000:01:00.0: bar0 len 0x4000, start 0x1f00410000, end 0x1f00413fff, flags, 0x40200
[    2.146719] rp1 0000:01:00.0: bar1 len 0x400000, start 0x1f00000000, end 0x1f003fffff, flags, 0x40200
[    2.155981] rp1 0000:01:00.0: enabling device (0000 -> 0002)
[    2.162270] rp1 0000:01:00.0: chip_id 0x20001927
[    2.170959] genirq: irq_chip rp1_irq_chip did not update eff. affinity mask of irq 100
[    5.009561] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[    5.237119] rp1-pio 1f00178000.pio: Created instance as pio0
pi@moode:~ $

PCIe controller: Pimoroni NVMe Base for Raspberry Pi 5
https://www.adafruit.com/product/5845?gad_source=1&gbraid=0AAAAADx9JvT2Uc1scGbd8pfLnNHlqrIyp&gclid=EAIaIQobChMIz8PE47P6iwMVo15_AB2ZfhpaEAQYASABEgKY__D_BwE

NVMe stick: Crucial P3 1TB PCIe Gen3 3D NAND NVMe M.2 SSD
https://www.bhphotovideo.com/c/product/1710645-REG/crucial_ct1000p3ssd8_1tb_p3_nvme_pcie.html/?ap=y&ap=y&smp=y&smp=y&store=420&lsft=BI%3A5451&gad_source=1&gbraid=0AAAAAD7yMh0CnuDG9lQosL1qx1hNsryh6&gclid=EAIaIQobChMIiaSZwLT6iwMVHDStBh17LgH0EAQYASABEgLYhvD_BwE

pelwell · 2025-03-08T20:09:36Z

It's possible that the presence of the NVME drive changes the timing of the boot procedure. Between 6.6.74 and 6.6.75 the RP1 PIO and firmware drivers were improved to cope with arbitrary startup order. I can't tell whether or not the kernel you have includes that patch, but you can ensure that it does by running sudo rpi-update 1705add.

moodeaudio · 2025-03-08T21:05:38Z

Makes sense. The kernel we use is the official release package so not patched.