Multi-cam setup in the wild : Elephants

[kommanderpi]

Hi @mprib

Thanks for your awesome repo! Im hoping it might be a way forward with my ongoing research -

TLDR: I want to track elephant behavior using multi-cam setups (trail-cameras - sadly have to manually synchronize the time stamps)

I have a pose-estimation model that performs great (PachySet! - fined tuned deeplabcut superquadruped mode).

The problem lies in calibrating multiple cameras for 3d pose-estimation - I have attached an image for reference:

The image depicts two different potential setups - one with 24 cameras, the other with 28 cameras

Pink = camera coverage
Black lines = camera views
Red blocks = static objects in the scene
For scale: the red blocks are 15m apart.

Do you have any thoughts about how I might go about calibrating the cameras? I am skeptical about the chacuro/checkerboard method - but I imagine there might be a method using landmarks in the scene (static markers that I can get gps coordinates of and know relative positions of - painted steel rods or traffic cones or rubber duckies)? Perhaps the static objects could be the markers?

I understand that i could use one (or more cameras as anchors for longer arrays - for example the above and below the static objects?

Thank you so much!

[mprib]

Hello @kommanderpi,

Thank you for reaching out! Your plans for capturing elephant behavior sound fascinating, and I'm happy to offer some thoughts on the challenges you've outlined. I've broken down my recommendations into a few key areas.

Camera Calibration: A Two-Stage Process

It's helpful to think about camera calibration in two distinct stages: intrinsic and extrinsic.

• Intrinsic Calibration (Lens Properties):
  This stage involves determining the focal length and lens distortion of each camera. I recommend using a standard Charuco board for this. This process only needs to be performed once for each camera and can be done in a controlled environment, like on a desk, before you set them up in the field. You would capture footage of the board with each camera individually to determine these parameters.

• Extrinsic Calibration (Camera Positions):
  This is the more complex part, where you determine the 3D position and orientation of each camera relative to the others. Using static markers is a viable approach, though Caliscope isn't set up for that out of the box. Instead of objects that are hard to precisely locate, I would suggest using large, single Aruco markers. These can be printed at a scale that will be clearly visible from a distance and positioned in the environment. Using Aruco markers would provide the high-accuracy common points needed across different camera views. To improve the robustness of the extrinsic calibration, you could also consider moving a large Aruco marker through the capture volume. This would generate a greater number of unique points for the calibration algorithm to work with. However, this could introduce motion blur, which can be a significant problem when tracking corners at a sub-pixel level.

A Critical Prerequisite: Time Synchronization

I want to highlight a major hurdle that you will need to address before you can achieve accurate 3D triangulation: time synchronization.

In my experience, this is a non-trivial problem. Even with cameras started simultaneously, their internal clocks will drift over time, and this will need to be corrected for frequently. If the video streams are out of sync, the triangulation results can become highly inaccurate very quickly.

I would strongly recommend focusing on developing a robust method for time-synchronizing your trail cameras first. Once you have a reliable way to ensure your footage is synchronized, you will be in a much better position to tackle the calibration and 3D reconstruction.

I am currently in the final stages of my dissertation defense, so my availability is quite limited for the next couple of weeks. However, I am very interested in your project and would definitely be interested in an ongoing discussion to see what might be possible or offer whatever help I can.

Regards,

Mac