SparkFlex timeouts shortly after startup using Spark template

[tbeadle]

Describe the question you have.
We just started using the Spark template and have noticed that after a few seconds (maybe 10-15 or so), some of the Spark Flex controllers start timing out in the SparkOdometryThread when calling getPosition(). I've confirmed that by logging the result of getLastError(), which is kTimeout. Once they get into that state, it seems to keep going and going, so once it starts, we never get any more odometry information because isValid is false. Do you have any suggestions as to why this might be happening. We have the Spark Max's for the turn motors and the Spark Flex's for the drive motors on the CAN bus connected directly to the RoboRIO (no CANIVORE or anything like that). There are no other devices on the CAN bus besides the PDP. Thank you.

[jwbonner]

Can you send the AdvantageKit log file from a time that this happened? That may be able to help with tracking down the issue.

[tbeadle]

Here you go!

akit_25-01-17_18-27-11.zip

You'll see, if you look at the Drive/Gyro/YawPosition/value or Drive/Module#/DrivePositionRad that the robot is rotating, first one way and then the other. But at about 26 seconds, the Drive/Gyro/OdometryYawPositions/0/value and Drive/Module0/OdometryDrivePositionsRad/0 stop changing while the non-odometry values continue to change as I rotate the robot. This is also the time when we get an error on the console "[CAN SPARK] IDs: 3, timed out while waiting for Periodic Status 2".

Thanks for taking a look.

[tbeadle]

FWIW, It's also about 26.4 seconds where /Drive/Module2/DriveConnected goes to false for a time.

[tbeadle]

I tried logging RobotController.getCANStatus().percentBusUtilization once a second and it seems reasonable to me. It hovers around 27%. Occasionally, it will jump to about 45% for a second and then right back down.

[tbeadle]

We just tried using the MAXSwerve Java Template and added URCL logging. In AdvantageScope, we tracked getPosition of the drive motors and it seems to consistently get values, albeit on the normal robot iteration frequency.

[jwbonner]

Based on the URCL data, it appears that the front right drive motor was genuinely disconnected for roughly ~1s just as the robot is stopping the turn. The fact that this lines up with what appears to be a hard acceleration certainly suggests that the root cause is some kind of wiring or power issue on that motor.

In addition to the genuine disconnect, I think the key issue is why the odometry thread continues to produce errors when reading data. I don't see anything in the template that is inconsistent with the documented usage of the APIs, so this is most likely related to an issue in REVLib. You might want to experiment with the setCANTimeout method on the Spark, though it's not clear whether that is supposed to affect getter calls. Otherwise, I think you'll have the most success by reaching out to REV's support email and see if they can shed any light on what might be happening that causes a device to continue returning kTimeout after a brief disconnect.

[jwbonner]

It looks like there has been some discussion of this issue with REV in this thread: https://www.chiefdelphi.com/t/rev-spark-flex-can-timeout-periodic-status-2/480555?u=jonahb55

It seems like this is ultimately a firmware or library issue for the Spark Flex. Disabling the error check in the odometry thread is a temporary workaround, but is problematic because you will still get invalid data occasionally that will not be properly filtered.

[tbeadle]

Thanks so much for taking a look and the link to that CD thread. I will definitely be watching that closely.