Project

Back to overview

Modeling Deformable 3-D Surfaces from Video

English title Modeling Deformable 3-D Surfaces from Video
Applicant Fua Pascal
Number 137525
Funding scheme Project funding (Div. I-III)
Research institution Laboratoire de vision par ordinateur EPFL - IC - ISIM - CVLAB
Institution of higher education EPF Lausanne - EPFL
Main discipline Information Technology
Start/End 01.01.2012 - 31.12.2013
Approved amount 103'290.00
Show all

Keywords (3)

Surface Modeling ; Deformable Models; 3D Reconstruction

Lay Summary (English)

Lead
Lay summary

Being able to recover the 3D shape of deformable surfaces using a single camera would make it possible to field reconstruction systems that run on widely available hardware. They could perform many important tasks, ranging from accurate monitoring of non-rigid structures to modeling organ deformations during endoscopic surgery and designing special effects for entertainment purposes. However, because many different 3D shapes can have virtually the same projection, such monocular shape recovery is inherently ambiguous.

The solutions that have been proposed over the years mainly fall into two classes: Those that involve physics-inspired models and those that rely on a non-rigid structure-from-motion approach. The former solutions often entail designing complex objective functions and require hard-to-obtain knowledge about the precise material properties of the target surfaces. The latter depend on points being reliably tracked in image sequences and are only effective for relatively small deformations.

In earlier phases of this project, we showed that simply constraining the distances between selected surface points to remain constant is enough to recover 3D shape from a single input image, provided that point correspondences can be established with a reference image in which the shape is known. Furthermore, the shape-recovery problem can be formulated as a convex problem that can be solved using standard mathematical routines and does not require an initial shape estimate.

Our algorithms are very reliable when enough correspondences can be established between the reference image and the input image. Unfortunately, this is not always the case for example because the surfaces are not sufficiently textured or because the texture is repetitive. In the continuation of this project, we will therefore to extend our method as follows.

- We will exploit additional sources of image information such as shading, texture, and contours.

- We will develop practical approaches to simultaneously establishing correspondences and recovering shape to handle situations, such as when the textures are repetitive, that make it ineffective to dissociate the two steps.

- We will tailor our deformation models for specific surface materials and learn their parameters online.

Bringing together these three strands of research will result in 3--D surface reconstruction algorithms that are more robust, more accurate, and much easier to deploy in real-world applications.

Direct link to Lay Summary Last update: 21.02.2013

Responsible applicant and co-applicants

Employees

Name Institute

Publications

Publication
A Constrained Latent Variable Model
Varol Aydin, Salzmann Mathieu, Fua Pascal, Urtasun Raquel (2012), A Constrained Latent Variable Model, in Conference on Computer Vision and Pattern Recognition.
Laplacian Meshes for Monocular 3D Shape Recovery
Ostlund Jonas, Ngo Dat, Varol Aydin, Fua Pascal (2012), Laplacian Meshes for Monocular 3D Shape Recovery, in European Conference on Computer Visio.

Collaboration

Group / person Country
Types of collaboration
Toyota Technological Institute United States of America (North America)
- Publication
- Exchange of personnel

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
European Conference on Computer Vision Poster 08.10.2012 Florence, Italy Ostlund Jonas;
Conference on Computer Vision and Pattern Recognition Talk given at a conference 18.06.2012 Providence, United States of America Ostlund Jonas;


Associated projects

Number Title Start Funding scheme
126524 Modeling Deformable 3-D Surfaces from Video 01.10.2009 Project funding (Div. I-III)
153121 Modeling Deformable 3-D Surfaces from Video 01.04.2014 Project funding (Div. I-III)

Abstract

Being able to recover the 3D shape of deformable surfaces using a single camera would make it possible to field reconstruction systems that run on widely available hardware. They could perform many important tasks, ranging from accurate monitoring of non-rigid structures to modeling organ deformations during endoscopic surgery and designing special effects for entertainment purposes. However, because many different 3D shapes can have virtually the same projection, such monocular shape recovery is inherently ambiguous.The solutions that have been proposed over the years mainly fall into two classes: Those that involve physics-inspired models and those that rely on a non-rigid structure-from-motion approach. The former solutions often entail designing complex objective functions and require hard-to-obtain knowledge about the precise material properties of the target surfaces. The latter depend on points being reliably tracked in image sequences and are only effective for relatively small deformations.In earlier phases of this project, we showed that simply constraining the distances between selected surface points to remain constant is enough to recover 3D shape from a single input image, provided that point correspondences can be established with a reference image in which the shape is known. Furthermore, the shape-recovery problem can be formulated as a convex problem that can be solved using standard mathematical routines and does not require an initial shape estimate.Our algorithms are very reliable when enough correspondences can be established between the reference image and the input image. Unfortunately, this is not always the case for example because the surfaces are not sufficiently textured or because the texture is repetitive. In the continuation of this project, we will therefore to extend our method as follows.- We will exploit additional sources of image information such as shading, texture, and contours.- We will develop practical approaches to simultaneously establishing correspondences and recovering shape to handle situations, such as when the textures are repetitive, that make it ineffective to dissociate the two steps.- We will tailor our deformation models for specific surface materials and learn their parameters online.Bringing together these three strands of research will result in 3--D surface reconstruction algorithms that are more robust, more accurate, and much easier to deploy in real-world applications.
-