The camera body is instrumented (left) for opto-tracking, which enables to know its pose at each frame acquisition instant (right).
3D representation of the scene
The system of coordinates of the LED markers is not coincident with the system of coordinates of the camera, and estimating the camera motion between frames requires knowing the constant rigid transformation that maps one reference frame into the other. The problem of determining this transformation X is called Hand-Eye Calibration.
Hand-Eye calibration scheme
The hand-eye calibration uses as inputs the motion A of the camera reference frame (the eye), typically computed from plane-to-image homographies, and the motion B of the camera body (the hand), that is measured by the opto-tracker. We propose in [B5] a novel approach for hand-eye calibration that is more robust and stable than the state-of-the-art, and is able to achieve accurate results from a minimum number of input motions.
Unfortunately, and despite of all our efforts, we arrived to the conclusions that the camera motion can not be accurately determined in an "open-loop manner" using exclusively opto-tracking measurements. This is shown in the figure below where it can be observed that the epipolar lines between two endoscopic views are not correctly aligned.
The reasons are two-fold:
(i) endoscopic images are acquired at very close range (1 to 2 cm). In these circumstances errors in the hand-eye calibration of 1mm correspond to several pixels in the image;
(ii) during operation the lens probe moves with respect to the camera body, which means that the rigid transformation between camera and markers reference frame is not strictly constant.
We are currently working in fusing visual odometry with the optical measurements to overcome this difficult and accurately recover the camera motion in real-time.