3D Reconstruction ; Deformable Models ; Surface Modeling; Computer Vision
Ngo D, ParkS, Jorstad A, Crivellaro A, Yoo C, Fua P (2015), Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture, in ICCV
, Santiago, Chile.
Magnenat Stephane, Ngo Dat, Zund Fabio, Ryffel Mattia, Noris Gioacchino, Rothlin Gerhard, Marra Alessia, Nitti Maurizio, Fua Pascal, Gross Markus, Live Texturing of Augmented Reality Characters from Colored Drawings, in IEEE Transactions on Visualization and Computer Graphics
Ngo Tien Dat, Östlund Jonas, Fua Pascal, Template-based Monocular 3D Shape Recovery using Laplacian Meshes, in IEEE Transactions on Pattern Analysis and Machine Intelligence
Being able to recover the 3D shape of deformable surfaces using a single camera will make it possible to field reconstruction systems that only require ordinary passive cameras, such as those that now equip most mobile devices. It will also make 3D shape recovery possible in more specialized contexts, such as when performing endoscopic surgery or using a fast camera to capture the deformations of a rapidly moving object. However, because many different 3D shapes can have virtually the same projection, such monocular shape recovery is inherently ambiguous.The solutions that have been proposed over the years mainly fall into two classes: Those that involve physics-inspired models and those that rely on a non-rigid structure-from-motion approach. The former solutions often entail designing complex objective functions and require hard-to-obtain knowledge about the precise material properties of the target surfaces. The latter depend on points being reliably tracked in image sequences and are only effective for relatively small deformations.To overcome these limitations, we have developed approaches that rely on establishing correspondences between in an input image in which the 3D shape is to be recovered and a reference image in which the 3D shape is known. We showed that 3D shape recovery under those conditions could be formulated as an under-constrained linear problem. Furthermore, either introducing inextensibility constraints or using control points to reduce the dimensionality of the problem turns it into a well-posed problem, which can be solved either in closed-form or using convex optimization.Our algorithms can now reliably handle deforming 3D surfaces but do not take their environment into account. This is a severe limitation because, in the real world, objects and surfaces do no exist in isolation. Instead, they interact with each other and modeling these interactions is key to accurate reconstruction and understanding of the physical phenomena at play. Examples include balls being hit by rackets, bats, and clubs at ballgames and organs being prodded by surgical tools during operations.This is the topic we intend to address in the continuation of this project and we will focus on the following two issues.- Modeling contact areas: When two objects come into contact, the contact area is hidden from the camera but should nevertheless be modeled properly to impose the right constraints on the 3D reconstruction and increase accuracy. We will therefore introduce consistency constraints that guarantee physically possible behavior of the contact area.- Explicitly modeling folds: Around contact points, surface texture often becomes difficult to exploit but shape information can be obtained from the surface folds that usually appear. We will therefore look into approaches to automatically detecting these folds and using them to impose appropriate differential constraints on the reconstructed surfaces.This will result in 3--D surface reconstruction algorithms that are more robust, more accurate, and can truly be deployed in real-world applications involving objects that interact with each other.