Given two or more calibrated views and a 3D model of an object present in the scene (prior knowledge), we aim to register the object with respect to the image acquisition viewpoints. There are two strategies for addressing the problem:
a) Recover 3D information from the views and perform some kind of 3D/3D registration
b) Perform directly 2D/3D registration, where the 3D model will somehow help on the recovery of the relevant structures
Why Is The Problem Difficult?:
In our case the objective is to reconstruct the bone structure from an endoscopic image sequences and register the result with a pre-operative 3D model of the patient's anatomy (e.g. a CT-scan of the knee).
Endoscopic images contain many problematic situations for stereo reconstruction and registration that are not fully solved by the algorithms described in the literature.
a) The Reconstruction Problem: Stereo reconstruction is specially difficult in an arthroscopic context because: (i) The surfaces in the scene are often textureless (e.g. bone); (ii) Since the images are acquired at very close range there are strong changes in viewpoint (wide-baseline stereo); (iii) there are often non-lambertian reflections and/or specularities; and (iv) the 3D reconstruction must be carried in real-time in order to be useful for surgical navigation purposes.
b) The Registration Problem: Registration is challenging in an arthroscopic context because: (i) it is difficult to segment the relevant structures, since the scene contains a lot of spurious stuff e.g. blood, ligaments residues; (ii) the endoscope provides a very partial view of the much larger 3D structure we aim to register (e.g. bone), and (iii) the endoscopic images are noisy.
Stereo From Induced Symmetry:
We propose a new cue to stereo, dubbed SymStereo, which can be considered as lying in between local and global stereo methods. Given a calibrated stereo pair, we show how to reconstruct the curve where a virtual cut plane intersects the scene structure
SymStereo profile cut
The result is a profile cut that resembles the one that would be obtained using a Laser Range Finder (LRF). While standard stereo relies on photo-consistency for matching pixels, we detect the image of the profile cut using signal symmetry analysis. This is possible because, since the virtual plane is assumed to pass between the cameras, then, the homography mapping of one view into the other gives raise to a warped image that is mirrored with respect to the profile cut projection. An important advantage of this symmetry cue is its global character that enables matching textureless image regions where photo-consistency measurements are ambiguous.
Resonstruction of Profile Cuts:
An example of stereo reconstruction using SymStereo from two endoscopic images of a bone is shown in the video.
Reconstruction result with SymStereo using two images
In the future we plan to estimate profile cuts from endoscopic images using SymStereo, that is robust to the difficulties posed by endoscopic sequences (low texture, specularities, and significant changes in viewpoint). The registration will be accomplished by aligning sparsely reconstructed profile cuts with the pre-operative 3D model.