Back to overview

Modeling Deformable 3-D Surfaces from Video

English title Modeling Deformable 3-D Surfaces from Video
Applicant Fua Pascal
Number 153121
Funding scheme Project funding
Research institution Laboratoire de vision par ordinateur EPFL - IC - ISIM - CVLAB
Institution of higher education EPF Lausanne - EPFL
Main discipline Information Technology
Start/End 01.04.2014 - 30.11.2015
Approved amount 89'319.00
Show all

Keywords (4)

3D Reconstruction ; Deformable Models ; Surface Modeling; Computer Vision

Lay Summary (French)

Pouvoir retrouver la forme 3D de surfaces déformables en n’utilisant qu’une seule caméra permettrait de le faire non seulement avec toutes celles qui équipent maintenant tous les téléphones portables et tablettes mobiles mais aussi dans des contextes plus spécialisés comme celui de la chirurgie endoscopique. Malheureusement c’est un problème difficile parce que beaucoup de formes 3D peuvent se projeter dans les images de manière très similaire. Dans ce projets, nous nous attachons à résoudre les ambiguïtés qui en résultent.
Lay summary

Pour résoudre le problème de la reconstruction monoculaire de surfaces déformables, nous avons développé des approches qui reposent sur ??l'établissement de correspondances entre dans une image d'entrée dans laquelle la forme 3D doit être retrouvée et une image de référence dans laquelle elle est connue. Nos algorithmes peuvent désormais gérer de manière fiable la déformation des surfaces 3D, mais ne prennent  pas en compte leur environnement. Il s'agit d'une limitation sévère parce que, dans le monde réel  les objets et les surfaces n’existent pas indépendamment les uns des autres. Au contraire, ils interagissent et la modélisation de ces interactions est essentielle pour la reconstruction précise et la compréhension des phénomènes physiques mis en jeu.

Ce sont ces interactions que nous allons étudier dans ce projet. Nous nous concentrerons sur la modélisation des zones de contact, par exemple dans le cas d’une balle frappée par une raquette de tennis, une batte de baseball, ou un club de golf et aussi dans celui d’un organe déformé par des instruments chirurgicaux au cours d’une opération.

Direct link to Lay Summary Last update: 28.03.2014

Responsible applicant and co-applicants


Name Institute


Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture
Ngo D, ParkS, Jorstad A, Crivellaro A, Yoo C, Fua P (2015), Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture, in ICCV, Santiago, Chile.
Live Texturing of Augmented Reality Characters from Colored Drawings
Magnenat Stephane, Ngo Dat, Zund Fabio, Ryffel Mattia, Noris Gioacchino, Rothlin Gerhard, Marra Alessia, Nitti Maurizio, Fua Pascal, Gross Markus, Live Texturing of Augmented Reality Characters from Colored Drawings, in IEEE Transactions on Visualization and Computer Graphics .
Template-based Monocular 3D Shape Recovery using Laplacian Meshes
Ngo Tien Dat, Östlund Jonas, Fua Pascal, Template-based Monocular 3D Shape Recovery using Laplacian Meshes, in IEEE Transactions on Pattern Analysis and Machine Intelligence .


Group / person Country
Types of collaboration
Disney Research Zürich Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
Washington State University United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Research Infrastructure
- Exchange of personnel


Title Year
Best Paper Honorable Mention at the International Symposium on Mixed and Augmented Reality, Fukukoa, Japan 2015

Associated projects

Number Title Start Funding scheme
163461 Modeling Deformable 3-D Surfaces from Video 01.09.2016 Project funding
126524 Modeling Deformable 3-D Surfaces from Video 01.10.2009 Project funding
137525 Modeling Deformable 3-D Surfaces from Video 01.01.2012 Project funding


Being able to recover the 3D shape of deformable surfaces using a single camera will make it possible to field reconstruction systems that only require ordinary passive cameras, such as those that now equip most mobile devices. It will also make 3D shape recovery possible in more specialized contexts, such as when performing endoscopic surgery or using a fast camera to capture the deformations of a rapidly moving object. However, because many different 3D shapes can have virtually the same projection, such monocular shape recovery is inherently ambiguous.The solutions that have been proposed over the years mainly fall into two classes: Those that involve physics-inspired models and those that rely on a non-rigid structure-from-motion approach. The former solutions often entail designing complex objective functions and require hard-to-obtain knowledge about the precise material properties of the target surfaces. The latter depend on points being reliably tracked in image sequences and are only effective for relatively small deformations.To overcome these limitations, we have developed approaches that rely on establishing correspondences between in an input image in which the 3D shape is to be recovered and a reference image in which the 3D shape is known. We showed that 3D shape recovery under those conditions could be formulated as an under-constrained linear problem. Furthermore, either introducing inextensibility constraints or using control points to reduce the dimensionality of the problem turns it into a well-posed problem, which can be solved either in closed-form or using convex optimization.Our algorithms can now reliably handle deforming 3D surfaces but do not take their environment into account. This is a severe limitation because, in the real world, objects and surfaces do no exist in isolation. Instead, they interact with each other and modeling these interactions is key to accurate reconstruction and understanding of the physical phenomena at play. Examples include balls being hit by rackets, bats, and clubs at ballgames and organs being prodded by surgical tools during operations.This is the topic we intend to address in the continuation of this project and we will focus on the following two issues.- Modeling contact areas: When two objects come into contact, the contact area is hidden from the camera but should nevertheless be modeled properly to impose the right constraints on the 3D reconstruction and increase accuracy. We will therefore introduce consistency constraints that guarantee physically possible behavior of the contact area.- Explicitly modeling folds: Around contact points, surface texture often becomes difficult to exploit but shape information can be obtained from the surface folds that usually appear. We will therefore look into approaches to automatically detecting these folds and using them to impose appropriate differential constraints on the reconstructed surfaces.This will result in 3--D surface reconstruction algorithms that are more robust, more accurate, and can truly be deployed in real-world applications involving objects that interact with each other.