Project

Back to overview

3D Geometry and Motion Reconstruction from Incomplete Time-varying Data

English title 3D Geometry and Motion Reconstruction from Incomplete Time-varying Data
Applicant Gross Markus
Number 124738
Funding scheme Project funding (Div. I-III)
Research institution Laboratoire d'informatique graphique et géométrique EPFL - IC - ISIM - LGG
Institution of higher education ETH Zurich - ETHZ
Main discipline Information Technology
Start/End 01.10.2009 - 30.09.2010
Approved amount 53'300.00
Show all

Keywords (8)

3D acquisition; dynamic geometry; surface reconstruction; geometry; motion; 3D reconstruction; real-time scanning; depth video

Lay Summary (English)

Lead
Lay summary
Accurate digital reconstruction of complex 3D objects in motion is one of the central challenges in geometry processing and computer vision. The goal of this project is to develop a novel dynamic shape reconstruction framework that supports the digitization of arbitrary moving and deforming objects while avoiding complex acquisition setups or substantial user assistance. In order to achieve this goal we propose the development of a new geometry synthesis algorithm that establishes spatio-temporal correspondences across input scans to aggregate geometric detail information, as well as appropriate motion vectors. This avoids the complexity of building a high-resolution template model, since all the geometric detail information is computed directly from the depth scans. Instead, we propose the use of a coarse template that can be selected from a database or quickly sketched by the user. With this minimal prior information, we sidestep the difficult problem of reconstructing the correct shape topology, since all the required topological information can be deduced from the coarse template shape. Our system is based on a simple, yet efficient optical 3D scanning system that can be assembled from off-the-shelf-components. Avoiding specialized hardware or complex studio setups enables the deployment of our dynamic 3D reconstruction pipeline to a wide variety of application settings, including entertainment, telepresence, education, digital archiving, medicine, or engineering. Accurate 3D geometry and motion data of human individuals can support medical diagnosis or enable the design of advanced prosthetics. Biologists will have a powerful new tool for studying animals and complex ecosystems. Engineers will be able to accurately measure the 3D geometry of dynamic systems and machinery in motion. In addition, academics and industry related to geometry processing will benefit from our research. The core algorithmic advances of the proposed project, a dynamic deformation model, a global optimization method for spatio-temporal correspondence computation, and a novel geometry representation tailored to dynamic shapes, will be relevant in other application settings, such as computer animation, physical simulation, and human-computer interaction.
Direct link to Lay Summary Last update: 21.02.2013

Responsible applicant and co-applicants

Employees

Associated projects

Number Title Start Funding scheme
129607 Realtime Acquisition and Dynamic Modeling of Human Faces, Upper-Bodies, and Hands (D-A-CH/LAV) 01.10.2010 Project funding (special)
112122 Spatio-Temporal Registration of Depth Video 01.07.2006 Project funding (Div. I-III)

Abstract

Since the invention of the first practical optical camera in the early 19th century, the acquisition of real-world objects has become an integral part of our daily life. Visual sensing technologies have seen three waves of evolution so far: images, videos, and more recently, 3D geometry. Similar to the transition from static images to dynamic video, we are currently faced with the challenge to acquire accurate digital representations of complex 3D models in motion.This project is a continuation of SNF project "Spatio-Temporal Registration of Depth Video" and is concerned with the development and implementation of a new framework for digital 3D geometry reconstruction from incomplete, time-varying data. Our goal is to reconstruct a full 3D representation of a moving and deforming object using a novel optical acquisition system that provides high-speed digitization from a single viewpoint. The main challenge in animation reconstruction arises from the fact that the scanner only acquires partial data due to occlusion. Since the scanned object is constantly in motion, data collected from different parts of the object at different time instances cannot simply be merged as in existing scanning systems for rigid objects that maintain their shape during the scanning process. To address this challenge, current state-of-the-art techniques assume that a highly detailed model has been digitized in a static pose prior to scanning the shape in motion. During the actual dynamic acquisition, this high-resolution model is then warped to match the deformation estimated from the captured data. However, building such a static template is a cumbersome and costly process that typically requires a complex acquisition setup and substantial manual intervention. In addition, the applicability of the method is limited to objects that can actively cooperate during digitization by holding still for an extended period of time. This requirement is not satisfied in many acquisition settings, e.g. when scanning an infant or an animal, or when the scanning process needs to be non-intrusive.Our goal is to lift these restrictions and develop a novel dynamic shape reconstruction framework that supports the digitization of arbitrary moving and deforming objects while avoiding complex acquisition setups or substantial user assistance. In order to achieve this goal we propose the development of a new geometry synthesis algorithm that establishes spatio-temporal correspondences across input scans to aggregate geometric detail information, as well as appropriate motion vectors. This avoids the complexity of building a high-resolution template model, since all the geometric detail information is computed directly from the depth scans. Instead, we propose the use of a coarse template that can be selected from a database or quickly sketched by the user. With this minimal prior information, we sidestep the difficult problem of reconstructing the correct shape topology, since all the required topological information can be deduced from the coarse template shape.The proposed research builds upon our recent results, a novel deformation model and a global optimization scheme for pairwise non-rigid registration. A key contribution will be a re-formulation of this optimization scheme to multi-scan sequences and an extension of the deformation model that dynamically adapts to the deformation of the acquired shape. In addition, we plan to devise a new geometry representation that separates static geometric detail from dynamic detail that evolves over time. This enables a decoupling of motion and geometry and facilitates the use of specialized data structures that directly exploit the temporal and spatial redundancy in the acquired data. With these algorithms in place, our framework has the potential to replace the de facto standard technology for capturing motion based on tracking markers attached to the scanned object. Beyond the digitization of human actors for entertainment applications, applications in robotics and engineering, cultural heritage, education, or medicine can benefit from the proposed research.
-