Synchronization error with image timestamp

[francescoo99]

Hi everyone!
I'm currently working on an application in which I have a basler camera mounted on a robot end-effector, thus I need to synchronize the image timestamp with the robot end-effecor pose timestamp while the robot is moving. I tried to do it in two different ways:

1. Using the TimeSynchronizer from message_filters package. As from the documentation, messages are synchronized by their timestamps.
2. Using a custom code that pairs the received image with the end-effector pose with the closest timestamp.

In both cases synchronization did the job since the time difference I got from the synchronized couple image-pose was acceptable.
In my application, the synchronized couple of image and pose is recorded after a specific event happens (let's call them sync_pose and sync_img).
The problem is that when I manually move the robot in sync_pose and take the photo from pylonviewer, such a photo does not minimally coincide with sync_img. This is very strange since the ROS2 clock type is SYSTEM_TYPE for both controller and pylon driver. Moreover, even if there may be some delays in grabbing images, they should keep their timestamps and be coupled with and older end-effector pose.

Then I knew about pylon-instant-camera repo that publishes images at our maximum frequency of 168Hz (same frequency as pylonviewer) and after conducting the same test as above (with method 2) I got two comparable images when coming back to sync_pose.

Additional info:

• I am using raw images (bayer encoding);
• I get images at 130hz on the published topic by the driver (in pylonviewer is actually 168 Hz);
• I get poses at 250hz on the published topic by the controller;
• Camera model: a2A1920-165g5cBAS;
• I'm working on ROS 2 humble with Ubuntu 22 and
• I'm using pylon 7.4.0 SDK;
• We use 10Gbit ethernet port;
• MTU size is set to 9000.

I attach the configuration file I'm using here below:

default.txt