Project

Surface Modeling ; Deformable Models; 3D Reconstruction

Lead
Lay summary
Being able to recover the 3D shape of deformable surfaces using a single camera would make it possible to field reconstruction systems that run on widely available hardware. They could perform many important tasks, ranging from accurate monitoring of non-rigid structures to modeling organ deformations during endoscopic surgery and designing special effects for entertainment purposes. However, because many different 3D shapes can have virtually the same projection, such monocular shape recovery is inherently ambiguous. The solutions that have been proposed over the years mainly fall into two classes: Those that involve physics-inspired models and those that rely on a non-rigid structure-from-motion approach. The former solutions often entail designing complex objective functions and require hard-to-obtain knowledge about the precise material properties of the target surfaces. The latter depend on points being reliably tracked in image sequences and are only effective for relatively small deformations. In earlier phases of this project, we showed that simply constraining the distances between selected surface points to remain constant is enough to recover 3D shape from a single input image, provided that point correspondences can be established with a reference image in which the shape is known. Furthermore, the shape-recovery problem can be formulated as a convex problem that can be solved using standard mathematical routines and does not require an initial shape estimate. Our algorithms are very reliable when enough correspondences can be established between the reference image and the input image. Unfortunately, this is not always the case for example because the surfaces are not sufficiently textured or because the texture is repetitive. In the continuation of this project, we will therefore to extend our method as follows. - We will exploit additional sources of image information such as shading, texture, and contours. - We will develop practical approaches to simultaneously establishing correspondences and recovering shape to handle situations, such as when the textures are repetitive, that make it ineffective to dissociate the two steps. - We will tailor our deformation models for specific surface materials and learn their parameters online. Bringing together these three strands of research will result in 3--D surface reconstruction algorithms that are more robust, more accurate, and much easier to deploy in real-world applications.

Direct link to Lay Summary

Last update: 21.02.2013

Active participation

Number	Title	Start	Funding scheme

Being able to recover the 3D shape of deformable surfaces using a single camera would make it possible to field reconstruction systems that run on widely available hardware. They could perform many important tasks, ranging from accurate monitoring of non-rigid structures to modeling organ deformations during endoscopic surgery and designing special effects for entertainment purposes. However, because many different 3D shapes can have virtually the same projection, such monocular shape recovery is inherently ambiguous.The solutions that have been proposed over the years mainly fall into two classes: Those that involve physics-inspired models and those that rely on a non-rigid structure-from-motion approach. The former solutions often entail designing complex objective functions and require hard-to-obtain knowledge about the precise material properties of the target surfaces. The latter depend on points being reliably tracked in image sequences and are only effective for relatively small deformations.In earlier phases of this project, we showed that simply constraining the distances between selected surface points to remain constant is enough to recover 3D shape from a single input image, provided that point correspondences can be established with a reference image in which the shape is known. Furthermore, the shape-recovery problem can be formulated as a convex problem that can be solved using standard mathematical routines and does not require an initial shape estimate.Our algorithms are very reliable when enough correspondences can be established between the reference image and the input image. Unfortunately, this is not always the case for example because the surfaces are not sufficiently textured or because the texture is repetitive. In the continuation of this project, we will therefore to extend our method as follows.- We will exploit additional sources of image information such as shading, texture, and contours.- We will develop practical approaches to simultaneously establishing correspondences and recovering shape to handle situations, such as when the textures are repetitive, that make it ineffective to dissociate the two steps.- We will tailor our deformation models for specific surface materials and learn their parameters online.Bringing together these three strands of research will result in 3--D surface reconstruction algorithms that are more robust, more accurate, and much easier to deploy in real-world applications.