Some weird crashes

[h4writer]

Due to reasons we are currently on bitcraze 2023-02 and I hope to find some pointers into debugging a case further. So I hope somebody can gives their thoughts.

What are we seeing.

The crazyflie (LH deck + light deck) flies very well. Is flying an automated path and suddenly falls out of the sky and is stuck. We sometimes have this happen on the ground, sometimes in the air. It just restarts and is stuck. We cannot find out why. Flashing seems to sometime help. But it is very unreliable to test.

My question.

• Is this something that rings a bell? A fixed bug and similar story ... Please tell!
• When building, do we need to keep X amount of memory free, to make sure that stack / dynamically allocated values work. What is recommended?
• Is there still an issue with a compiler bug or something (building on ubuntu 22.04)
• ...

[gemenerik]

Unless it is something with the power (e.g. cable breakage, check this), it is likely that you hit an assert somewhere in the firmware. Seeing as it sometimes happens on the ground, try hooking up a debug adapter and catch any asserts

[h4writer]

We do get the following in the logs:
../src/hal/src/radiolink.c:164

We have bought a debug adapter and we hope we can reproduce on the ground. We have not found a way to trigger it reliable yet.

[ataffanel]

This is the CRTP communication receive queue that is overflowing: https://github.com/bitcraze/crazyflie-firmware/blob/0f2f0f07d2c2463cb07e4b187f9a9a58b689d124/src/hal/src/radiolink.c#L164

This problem has unfortunatly been seen before quite a lot. There can be two related causes:'

• The Crazyflie CPU is overloaded, a task take too much time to run and so there is not enough time to empty the packet queue fast enough
• Too many packets are sent to the Crazyflie, this should not be possible though since the radio link is relativly show

If you are using an app, you could try to reduce its priority to see if it helps.

[h4writer]

Thanks for the pointers! That really helped out. I'm focusing now on the CPU overloading.

Giving the CPU overload, I could think of a few things. I was able to discard already some, by testing.
Though I have another hypotheses.

Could this be caused by increasing the CONFIG_DECK_LIGHTHOUSE_MAX_N_BS=16.
We are also using 4 lighthouses. I could be wrong, but I think this hasn't happened yet when Lighthouse wasn't active.
Did you or colleagues have had this sort of issues with more lighthouses?

[ataffanel]

Yes, the lighthouse configuration could be a cause. You can try to dial back the max BS to 4. We have been flying with 4 BS without major problem though so this should work.

I understand you cannot use the latest firmware, but I would suggest trying with it though to find if the problem has been solved. Lighthouse is something we have been working quite a lot on in the last years so it is possible this problem will not happen with the latest version. If updating the firmware is really impractical, you could try to back-port the lighthouse related code on your firmware.