Tkach Anastasia, Tagliassachi Andrea, Pauly Mark (2016), Sphere-meshes for real-time hand modeling and tracking, in ACM Transactions on Graphics (TOG)
, (6), 222.
Tagliasacchi Andrea, Schroeder Matthias, Tkach Anastasia, Bouaziz Sofien, Botsch Mario, Pauly Mark (2015), Robust articulated-ICP for real-time hand tracking, in SGP '15 Proceedings of the Eurographics Symposium on Geometry Processing
, (5), 101-114.
Ichim Alexandru, Bouaziz Sofien, Pauly Mark (2015), Dynamic 3D avatar creation from hand-held video input, in ACM Transactions on Graphics (TOG)
, (4), 45.
Remelli Edoardo, Tkach Anastasia, Tagliasacchi Andrea, Pauly Mark, Low-Dimensionality Calibration through Local Anisotropic Scaling for Robust Hand Model Personalization, in International Conference on Computer Vision (ICCV), 2017
, IEEE, Venice.
Tkach Anastasia, Tagliassacchi Andrea, Remelli Edoardo, Pauly Mark, Fitzgibbon Andrew, Online Generative Model Personalization for Hand Tracking, in ACM Transactions on Graphics
Tracking and animating humans in motion is a fundamental problem in computer graphics and computer vision. A particularly important question is how to accurately reconstruct the shape and articulation of human hands. Hand motion is a crucial component of non-verbal communication, plays an important role in the animation of humanoid avatars, and is central for numerous human-computer interfaces. While accurate realtime body and face tracking systems have been proposed in recent years, hand tracking solutions of comparable quality are still lacking, in particular ones that can be deployed in the context of consumer applications.In this project we plan to develop new methods and algorithms for realtime hand tracking and animation with a particular focus on accuracy and ease-of-use. We aim to faithfully reconstruct intricate hand geometry and motion with an acquisition system that can be readily deployed in consumer-level applications. Our goal is to integrate the developed algorithms with existing face tracking technology to obtain a complete system for online human communication in a desktop environment. This system will allow an entirely new form of realtime interactions based on digital avatars, with numerous applications in digital content creation, online interaction and gaming, education and training, or human computer interfaces.Accurate hand tracking with a non-invasive sensing device in realtime is a highly challenging scientific prob- lem. Human hands are very articulate and therefore require models with sufficiently many degrees of freedom to adequately describe the corresponding motion space. Hand motion is often fast and complex, exhibiting intricate geometric configurations and complicated contact patterns among fingers. In addition, our focus on a single sensor setup to facilitate wide-spread applicability leads to ambiguous and often incomplete input data, caused by significant self-occlusions of the tracked hands.All these aspects mandate new algorithmic solutions. We will build on our extensive experience in face tracking, geometry optimization, and realtime dynamic modeling to investigate these problems and develop new algorithms for realtime human hand tracking. We will start our investigations by developing an offline system for accurate and robust hand-tracking, using pre-build dynamic hand models tailored to a specific user. We then successively refine our solution towards higher performance and higher usability. This requires new efficient algorithms to meet the realtime goal and sophisticated pre-processing and online learning methods to achieve maximal usability without requiring any additional calibration or user assistance. We plan to integrate and improve state-of-the-art robust optimization methods and machine learning algorithms that allow exploiting the strong spatial and temporal coupling of hand articulation. We will also take advantage of recent advances in commodity RGB-D sensing devices that allow concurrent acquisition of geometry and texture. A novel registration method that combines articulated 3D alignment with high-quality optical flow will be specifically designed for hand geometry and texture.In the final stage of the project, we will integrate the developed hand tracking solution with our existing face tracking algorithms to obtain a complete system for realtime acquisition and animation of human upper bodies, hands, and faces. This system will enable new forms human interaction in a desktop environment, where users communicate through virtual avatars in realtime. With the increasing proliferation of commodity sensors, we expect our research to facilitate numerous new applications in online communication, with profound potential impact on society.