Exception on startup after the arm was running for a long time: timeout detected: BaseClient::SetServoingMode

[schornakj]

Robot: Kinova Gen3 Lite

I had to power cycle the arm to resolve this.

kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] terminate called after throwing an instance of 'std::runtime_error'                                                                       
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1]   what():  timeout detected: BaseClient::SetServoingMode
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1]                                 
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] Stack trace (most recent call last):                            
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #18   Object "", at 0xffffffffffffffff, in 
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #17   Object "/opt/ros/humble/lib/controller_manager/ros2_control_node", at 0x55ca72c8bd34, in            
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #16   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f9681baae3f, in __libc_start_main               
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #15   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f9681baad8f, in                                                                 kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #14   Object "/opt/ros/humble/lib/controller_manager/ros2_control_node", at 0x55ca72c8b865, in                                       
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #13   Object "/opt/ros/humble/lib/libcontroller_manager.so", at 0x7f968225dfbb, in controller_manager::ControllerManager::ControllerManager(std::shared_ptr<rclcpp::Executor>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&
)                                                                                                                                                                                                                  kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #12   Object "/opt/ros/humble/lib/libcontroller_manager.so", at 0x7f968225cff7, in controller_manager::ControllerManager::init_resource_ma
nager(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)                                                                                                                      kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #11   Object "/opt/ros/humble/lib/libhardware_interface.so", at 0x7f9681aaeaba, in hardware_interface::ResourceManager::load_urdf(std::__c
xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)                                                                                                                              kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #10   Object "/opt/ros/humble/lib/libhardware_interface.so", at 0x7f9681ad61db, in 
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #9    Object "/opt/ros/humble/lib/libhardware_interface.so", at 0x7f9681accdcb, in hardware_interface::System::initialize(hardware_interface::HardwareInfo const&)                                                                                                                                                                                           
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #8    Object "/opt/underlay_ws/install/kortex2_driver/lib/libkortex2_driver.so", at 0x7f965fd68e17, in kortex2_driver::KortexMultiInterfaceHardware::on_init(hardware_interface::HardwareInfo const&)                                                                                                                                                        
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #7    Object "/opt/underlay_ws/install/kortex2_driver/lib/libkortex2_driver.so", at 0x7f965fe76857, in Kinova::Api::Base::BaseClient::SetS
ervoingMode(Kinova::Api::Base::ServoingModeInformation const&, unsigned int, Kinova::Api::RouterClientSendOptions const&)                                                                                          kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #6    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f9681e77517, in __cxa_throw    
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #5    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f9681e772b6, in std::terminate()
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #4    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f9681e7724b, in                                                       kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #3    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7f9681e6bbbd, in 
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #2    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f9681ba97f2, in abort                                                           
kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #1    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f9681bc3475, in raise                                                           kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] #0    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f9681c17a7c, in pthread_kill                                                    kinova-rt-drivers-1            | [launcher_node-2] [ros2_control_node-1] Aborted (Signal sent by tkill() 105 0)  

[moriarty]

Here are the two calls to SetServoingMode in on_init

https://github.com/PickNikRobotics/ros2_kortex/blob/65f6d6502854cedac3caee1b345936cbca7d0101/kortex2_driver/src/hardware_interface.cpp#L197-L200

https://github.com/PickNikRobotics/ros2_kortex/blob/65f6d6502854cedac3caee1b345936cbca7d0101/kortex2_driver/src/hardware_interface.cpp#L216-L219

[schornakj]

The main takeaway I get from seeing these exceptions is that lots of calls to the Kortex API can throw exceptions, so we need to make sure we can catch and gracefully handle those.

[aalmrad]

Hello,

Apologies for the late reply. We are recently trying to tackle the github issues on our repositories more actively. We will implement additional code to properly handle the thrown exceptions.

Best,
Abed