RP2350 - Using `atomic_exchange` before `wfe` is triggering immediate wakeup

[jakoblell]

This issue is a followup to embassy-rs/embassy#4818, as far as I can tell this issue mainly affects the Embassy executor in Rust but it can also be reproduced with plain C/pico-sdk.

When user code is doing atomic operations (atomic_exchange in C or AtomicPtr.swap() in Rust) before a wfe in an idle loop, the wfe will immediately wake up instead of putting the device to sleep until the next real wakeup event (such as an interrupt).

In Embassy, the executor will use atomic operations to check if a task is pending - and if no task is pending it will sleep via wfe and check again after the next wakeup event (with the rationale that an interrupt can wake up tasks to be executed after the interrupt), see embassy-rs/embassy#4818 for a more detailed description with call flows. This process is not working as expected (that means not actually sleeping, thus keeping the CPU fully awake in a busy loop with increased power consumption) if the atomic operation itself is triggering a wakeup.

The following minimal C reproducer will demonstrate the issue (if you want to play around with that code, clone the pico-examples repository and put the code into https://github.com/raspberrypi/pico-examples/blob/master/hello_world/serial/hello_serial.c so that you can use the existing build system):

#include <stdio.h>
#include <stdatomic.h>
#include "pico/stdlib.h"

int main() {
    stdio_init_all();
    printf("Startup!\n");
    int i;
    atomic_int x = 42;
    for(i=0;i<1000;i++){
        int old = atomic_exchange(&x, 99);
        __asm__("wfe");
    }
    printf("After 1000x wfe!\n");
}

This program will print "After 1000x wfe!" right after "Startup!", which indicates that the wfe instruction is not actually putting the device to sleep. When commenting out the atomic_exchange line, it will hang indefinitely (meaning that the wfe is actually putting the device to sleep).

To find out whether this behavior is specific to the RP2350, I've also tried it on an STM32F411 and there the wfe is sleeping as expected even if there is an AtomicPtr.swap() right before it.

Since this issue occurs on the RP2350 but not on an STM32F411 and I didn't find a good explanation for this behavior, I came to the conclusion that this might be a bug/errata in the RP2350 chip - but since I'm not very familiar with the low-level details of the Arm Cortex-M architecture I might also be wrong on that. If this is intended/documented behavior, it would be great if someone could point me to the corresponding documentation.

[Wren6991]

For context, see: raspberrypi/pico-sdk#1812

The short version is: Cortex-M requires an event on loss of reservation. We also make the implementation-defined decision that a PE's successful exclusive store to its own reservation closes that reservation. The combination of these two behaviours is a useless event.

For future devices we're looking at changing our interpretation of the implementation-defined state transitions to remove the useless events in a way that is architecturally compliant.

It's worth reading that entire thread I linked for the full context.

[jakoblell]

Thanks for the quick response. The use case is a little bit different between Micropython and Embassy, for Embassy the executor loop has to use an atomic data structure in every loop iteration to check whether there are pending tasks. Do you have a recommendation on how this can be implemented in a safe way still allowing the CPU to enter wfe state without an immediate wakeup? Just clearing the pending event after checking for pending tasks is obviously not an option since there could be a race condition with real events coming in.

[Wren6991]

Instead of an atomic swap, are you able to use a different primitive which doesn't retire its reservation in the case where the work queue is empty? That should allow wfe to function normally.

Presumably that AtomicPtr operation is just being lowered to ldrex + strex anyway; if you've done an ldrex with the necessary orderings then this is already enough to decide if the work queue is empty, and in that case you can skip the strex and just sleep. In fact the opposite core writing to your reservation should wake you.