Back to overview

Be Your Avatar: Realtime Tracking and Animation for a Desktop Environment

English title Be Your Avatar: Realtime Tracking and Animation for a Desktop Environment
Applicant Pauly Mark
Number 153567
Funding scheme Project funding (Div. I-III)
Research institution Laboratoire d'informatique graphique et géométrique EPFL - IC - ISIM - LGG
Institution of higher education EPF Lausanne - EPFL
Main discipline Information Technology
Start/End 01.04.2014 - 30.06.2017
Approved amount 176'463.00
Show all

Keywords (3)

hand tracking; realtime tracking; animation

Lay Summary (German)

Das Verfolgen und Animieren von menschlichen Bewegungen mittels digitalen Techniken ist ein fundamentales Problem der Computergraphik und des computer-gestützten Bildverstehens. Eine besonders wichtige Frage ist die genaue Rekonstruktion des Form und Artikulation menschlicher Hände. Handbewegungen sind essenziell für nicht-verbale Kommunikation und spielen eine wichtige Rolle in der digitalen Animation von menschlichen Avataren sowie in der Mensch-Maschine Interaktion. In diesem Projekt werden wir neue Methoden und Algorithmen untersuchen, die das Erfassen und Verfolgen von komplexen Handbewegungen in Echtzeit ermöglichen.
Lay summary

Vorrangiges Ziel ist die genaue Erfassung menschlicher Handbewegungen mittels visueller Sensoren. Die aufgenommenen Handbewegungen sollen in Echtzeit auf ein digitales Modell übertragen werden, um neue Formen der Computer-Animation zu ermöglichen. Ein wichtiger Schritt ist die genaue Erfassung der Geometrie und des Bewegungsraums der menschlichen Hand. Dazu werden neueste 3D Scanning Methoden eingesetzt, die es erlauben, ein generisches Modell an die spezielle Handform eines individuellen Benutzers anzupassen. 
Diese Algorithmen werden mit einem bestehenden System zur Gesichtserkennung integriert, um eine vollständige Lösung für online Interaktion zu entwickeln. Dieses System erlaubt es als beliebiger Avatar in eine digitale Echtzeit-Kommunikation mit anderen Benutzern einzutreten.

Direct link to Lay Summary Last update: 30.03.2014

Responsible applicant and co-applicants



Sphere-meshes for real-time hand modeling and tracking
Tkach Anastasia, Tagliassachi Andrea, Pauly Mark (2016), Sphere-meshes for real-time hand modeling and tracking, in ACM Transactions on Graphics (TOG), (6), 222.
Robust articulated-ICP for real-time hand tracking
Tagliasacchi Andrea, Schroeder Matthias, Tkach Anastasia, Bouaziz Sofien, Botsch Mario, Pauly Mark (2015), Robust articulated-ICP for real-time hand tracking, in SGP '15 Proceedings of the Eurographics Symposium on Geometry Processing, (5), 101-114.
Dynamic 3D avatar creation from hand-held video input
Ichim Alexandru, Bouaziz Sofien, Pauly Mark (2015), Dynamic 3D avatar creation from hand-held video input, in ACM Transactions on Graphics (TOG) , (4), 45.
Low-Dimensionality Calibration through Local Anisotropic Scaling for Robust Hand Model Personalization
Remelli Edoardo, Tkach Anastasia, Tagliasacchi Andrea, Pauly Mark, Low-Dimensionality Calibration through Local Anisotropic Scaling for Robust Hand Model Personalization, in International Conference on Computer Vision (ICCV), 2017, IEEE, Venice.
Online Generative Model Personalization for Hand Tracking
Tkach Anastasia, Tagliassacchi Andrea, Remelli Edoardo, Pauly Mark, Fitzgibbon Andrew, Online Generative Model Personalization for Hand Tracking, in ACM Transactions on Graphics, (6).


Group / person Country
Types of collaboration
faceshift AG Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Research Infrastructure
Prof. Dr. Mario Botsch, University of Bielefeld Germany (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
ACM SIGGRAPH Asia Talk given at a conference Sphere-meshes for real-time hand modeling and tracking 23.11.2016 Macao, China Tkach Anastasia;
ACM SIGGRAPH Conference Talk given at a conference Dynamic 3D Avatar Creation from Hand-held Video Input 10.08.2016 Los Angeles, United States of America Ichim Alexandru;
Eurographics Symposium on Geometry Processing Talk given at a conference Robust articulated-ICP for real-time hand tracking 06.07.2015 Graz, Austria Pauly Mark; Bouaziz Sofien; Tkach Anastasia;


Title Year
Eurographics Annual Award for Best PhD Thesis - honorable mention 2019
Fritz Kutter Award 2016
Best Paper Award 2015

Associated projects

Number Title Start Funding scheme
129607 Realtime Acquisition and Dynamic Modeling of Human Faces, Upper-Bodies, and Hands (D-A-CH/LAV) 01.10.2010 Project funding (special)


Tracking and animating humans in motion is a fundamental problem in computer graphics and computer vision. A particularly important question is how to accurately reconstruct the shape and articulation of human hands. Hand motion is a crucial component of non-verbal communication, plays an important role in the animation of humanoid avatars, and is central for numerous human-computer interfaces. While accurate realtime body and face tracking systems have been proposed in recent years, hand tracking solutions of comparable quality are still lacking, in particular ones that can be deployed in the context of consumer applications.In this project we plan to develop new methods and algorithms for realtime hand tracking and animation with a particular focus on accuracy and ease-of-use. We aim to faithfully reconstruct intricate hand geometry and motion with an acquisition system that can be readily deployed in consumer-level applications. Our goal is to integrate the developed algorithms with existing face tracking technology to obtain a complete system for online human communication in a desktop environment. This system will allow an entirely new form of realtime interactions based on digital avatars, with numerous applications in digital content creation, online interaction and gaming, education and training, or human computer interfaces.Accurate hand tracking with a non-invasive sensing device in realtime is a highly challenging scientific prob- lem. Human hands are very articulate and therefore require models with sufficiently many degrees of freedom to adequately describe the corresponding motion space. Hand motion is often fast and complex, exhibiting intricate geometric configurations and complicated contact patterns among fingers. In addition, our focus on a single sensor setup to facilitate wide-spread applicability leads to ambiguous and often incomplete input data, caused by significant self-occlusions of the tracked hands.All these aspects mandate new algorithmic solutions. We will build on our extensive experience in face tracking, geometry optimization, and realtime dynamic modeling to investigate these problems and develop new algorithms for realtime human hand tracking. We will start our investigations by developing an offline system for accurate and robust hand-tracking, using pre-build dynamic hand models tailored to a specific user. We then successively refine our solution towards higher performance and higher usability. This requires new efficient algorithms to meet the realtime goal and sophisticated pre-processing and online learning methods to achieve maximal usability without requiring any additional calibration or user assistance. We plan to integrate and improve state-of-the-art robust optimization methods and machine learning algorithms that allow exploiting the strong spatial and temporal coupling of hand articulation. We will also take advantage of recent advances in commodity RGB-D sensing devices that allow concurrent acquisition of geometry and texture. A novel registration method that combines articulated 3D alignment with high-quality optical flow will be specifically designed for hand geometry and texture.In the final stage of the project, we will integrate the developed hand tracking solution with our existing face tracking algorithms to obtain a complete system for realtime acquisition and animation of human upper bodies, hands, and faces. This system will enable new forms human interaction in a desktop environment, where users communicate through virtual avatars in realtime. With the increasing proliferation of commodity sensors, we expect our research to facilitate numerous new applications in online communication, with profound potential impact on society.