MSP + LOG_DEBUG - complete FC lock-up (possible infinite loop)

[mstrakl]

Current Behavior

After losing 6 planes in fatal crashes, I finally found (≈99% certainty) a possible issue related to MSP communication combined with LOG_DEBUG usage. Using LOG_DEBUG statements together with other instances of MSP communication can lock up the flight controller completely, apparently stuck in an infinite loop. Result is a total freeze of the FC:

• Flight controls stop
• Telemetry stops
• FC cannot be disarmed
• Motors continue running in the last commanded state
• System appears completely dead

Steps to Reproduce

1. Add a LOG_DEBUG message (in some loop, where it's called periodically). In my case, it was in navigation_multicopter.c, function updatePositionVelocityController_MC. I called it once every 100 loops, to make sure not to overflow the buffer (but it seems that it does't matter for the issue in question).
2. Compile firmware, flash to FC.
3. Props off, power the FC, configure log debug reading, as instructed in: https://github.com/iNavFlight/inav/blob/master/docs/development/serial_printf_debugging.md
4. Connect with any kind of MSP reader, confirm that it's working by seeing the initialization LOG_DEBUG messages, of gyro and baro initializing
5. Wait for GPS fix, arm, activate POSHOLD so updatePositionVelocityController_MC is executed), and confirm LOG_DEBUG is being displayed
6. Now close the MSP reader instance
7. FC is now in locked up state (try disarming or moving controls, nothing happens - at least in my case).
8. Now start the MSP reader again - interestingly, FC restores normal run state, everything works, it disarms, and starts reacting to control inputs again, etc. -> to me this strongly indicates that it was in an infinite loop, not hard fault

Expected behavior

FC works normally, when MSP reader is stopped, and does not lock up or freeze

Suggested solution(s)

Check for infinite while loops (for example in msp_serial.c, or serial.c, or maybe somewhere else), which could lead to the described scenario, in case of buffer not clearing itself properly, or something similar.

Additional context

1. The same behavior can be triggered using Inav Configurator instead of the MSP reader. As soon as you close inav configurator, same thing happens.

• FC Board name and vendor: MATEK F405 Wing V2
• INAV version string: 9.0.1 (+ added LOG_DEBUG statement)

[daijoubu]

I think I might have found the issue. Let me try to recreate it. Serial debugging assumes there will always be a reader connected.

[daijoubu]

Can you send a link to your branch that you tested this on? I can't recreate it with my CAN debug messages on the F7 board and there aren't enough LOG_DEBUG calls in production branches.