Pinhole Camera Model Definition

[ErichZimmer]

@ronshnapp
When implementing camera models for tomographic PIV, I ran into your definition of the pinhole camera model. Typically, the pinhole model is defined along the lines as $P_w = K[ R | T ] × P_c'$ or some derivative of it like this. However, your implementation of the pinhole model seems quite unique. I went browsing a few academic databases and researchgate/Springer, but I could not find a similar definition. Can you elaborate on your decision to implement the calibration model in this way? Is it because of its ease of attaining a ray for a given pixel or the ray tracing algorithm based on voxels?

[ronshnapp]

Hi @ErichZimmer ,

So, the basics of the model I'm using are the same basic pinhole camera model people use: P_w = P_c \cdot [R] + O.
The difference that you see might originate from the addition of a quadratic error term. Essentially, the equation using the error terms is:
P_w = (P_c + Err) \cdot [R] + O
where Err is a quadratic function of the camera pixel locations. In the calibration process we first solve for the (errorless) linear transformations O and [R] and after that optimise it through the non linear terms.
The error term we used is not super common, though I wasn't the one who invented it. I decided to implement it this way because I thought that possibly it might be able to handle a diverse set of image distortions because we are essentially not specifying a model for the error, and also, if we found that the quadratic correction doesn't always work, we can very easily extend it to a cubic correction, etc. I wrote about the transformation we're using in the user manual, although not in too much details. If I remember correctly, the original work from which I took this error treatment is a report by Soren Ott from the early 2000's. I can search for it if you're interested.
Does this help answer your questions?

[ErichZimmer]

Hi Ron,

Thanks for pointing me to the right direction. I managed to implement the nonlinear error correction through polynomials, and it works quite well. For some reason, the radial and tangential distortion model (the one that was linked in the example implementation of a pinhole camera model) performs slightly better when there is only radial and tangential distortion. However, the polynomial distortion model seems to work better when using Scheimpflug mounts (this comparison included an additional distortion model on top of the radial and tangential distortion model for the OpenCV style distortion correction) and when there is mustache distortion.

[ronshnapp]

That's interesting. It could be nice to perform a thorough comparison between the models, perhaps also implementing a higher order correction (e.g. a third order polynomial).

[ErichZimmer]

It just so happens that I am working on a calibration and distortion model comparison. Currently, a 3rd order polynomial in x and y and 2nd order polynomial in z-axis performs the best while requiring a minimum of 19 calibration points. The pinhole model with radial and tangential distortion model performs 2nd best. Finally, pinhole model with 2nd order polynomial performs the worst. If I recount correctly, the RMSE for the example calibration data for camera 1 of the MyPTV example dataset was around 0.15, 0.20, and 0.25 respectively, +/- 0.05 due to memory issues. Just have to implement the rational polynomial model and pinhole model with rational distortion model and check for performance.

[ErichZimmer]

I re-ran the test for the MyPTV example dataset and here were the results:

Camera Model	RMSE Cam 1	RMSE Cam 2	RMSE Cam 3
Polynomial	0.17	0.35	0.37
Pinhole + R. & T.	0.20	0.37	0.40
Pinhole + 2nd order Poly	0.18	0.36	0.40

It is interesting to note for synthetic tests, the pinhole model with radial and tangential distortion correction is slightly more accurate than the pinhole model with polynomial distortion correction, while in real world tests, the differences are not significant (that is, as long as there are only radial and tangential distortion)

I'll implement the rational model(s) tonight and see how it performs.

[ErichZimmer]

Well, the 2nd order rational function has the same accuracy as the 2nd order poly, but required nonlinear optimization. So, I will stay with the poly model using NumPy lstsq to minimize the parameters. For inversion, I cheated and explicitly computed an inverse distortion matrix and used NumPy's listsq to minimize it just like the distortion correction matrix. The results are nearly the same as Pinhole + 2nd order poly. Thanks for helping me on this section of camera vision. :)

[ErichZimmer]

On a side note, how does you distortion function eta_zeta_from_Rbinv work to invert the distortion model? When I applied it to the normalized camera coordinates, it produced wildly incorrect results. Could this be because of your adaptation of the inversion to suite your camera model (e.g., using pixel coordinates instead of normalized camera coordinates or something to that effect)?

[ronshnapp]

Hi Erich. I would say that the inversion is in some sense Achilles' heel of this model, because the non-linearity means that you can't invert it analytically. To bypass the problem I linearized the correction term using a Taylor series. I have to say I never experienced a huge problem with it, so maybe it's worthwhile looking why it collapsed in your case?

[ErichZimmer]

Perhaps, implementing a 2nd or 3rd order rational polynomial distortion model may be worthwhile due to its possibly simple inversion. However, it requires more coefficients, a tradeoff that is worth it in my opinion.

[ErichZimmer]

For a reference for the pinhole camera model polynomial distortion correction, here is the link

[ErichZimmer]

@ronshnapp
I know this is a little off topic for an already solved discussion, but I was wondering about how you solved for a point between two rays analytically (link to function). I tried some epipolar geometry for an hour or so (albeit this was my first go at it), but I had a rough time since my camera model is practically the inverse of the model implemented in MyPTV. For instance, the origin values for my Tsai camera model implementation requires an inverse translation to mirror that of MyPTV (e.g., R' * -T). So, I am reverting back to reading articles on ray tracing and epipolar geometry which only got me so far for now. Hope it is not too much of a hassle for you to explain the theory behind your implementation for solving a point between to rays.

Kind Regards,
Erich

[ronshnapp]

Hi Erich,

I'm interested to hear what kind of a project you're working on. Perhaps having new alorthms implemented in MyPTV could be cool nice, if you're interested.

As for your question - there are two cases:

1. If two lines are crossing paths perfectly in 3D, then getting the point of intersection is very easy. You just equate the two curves and this gives you what you need. The problem is that this case almost never happens in real experiments due to the calibration and segmentation uncertainties, and this leads us to the second, more realistic method.

2. Here, instead of searching for an actual, precise crossing point, you are searching for two points along the 3D lines, P1 & P2, that the distance between them is the smallest. This is quite a well known optimisation problem that can be solved analytically (see for example this discussion in stackexchange: https://math.stackexchange.com/questions/2213165/find-shortest-distance-between-lines-in-3d). Once you have the two points, the crossing point of the two lines is approximated as the point that lies in between P1 and P2.

[ErichZimmer]

Hi Ron,

Thanks for your reply.

For the project I am working on, it is a volume self-calibration algorithm based on Wieneke's papers using particle segmentation and triangulation to lower the calibration error of a camera system for a tomographic PIV algorithm I am slowly working on. In essence, volume self-calibration has the capability to lower the misalignment between cameras to below 0.2 pixels, even if there were large misalignments of around 10 to 15 pixels. For the tomo-PIV algorithms to work, the calibration of the camera system must have a maximum error of around 0.4 pixels in order to attain reasonable results. Additionally, the particle information would be used to calibrate the elliptical gaussian weighting function for the multiplicative algebraic reconstruction technique (MART), which was shown to greatly improve the reconstruction quality.

See
Wieneke, B. (2008). Volume self-calibration for 3D particle image velocimetry. Experiments in Fluids, 45(4), 549–556. 10.1007/s00348-008-0521-5

Wieneke, B. (2018). Improvements for volume self-calibration. Measurement Science and Technology. 29. 10.1088/1361-6501/aacd45

[ronshnapp]

Hi @ErichZimmer , thanks for the explanation!
If you find it works for you, I'd be glad to hear and maybe implement also in MyPTV later on.