Back to overview

Heterogeneous Face Recognition

English title HFACE
Applicant Marcel Sébastien
Number 153554
Funding scheme Project funding (Div. I-III)
Research institution IDIAP Institut de Recherche
Institution of higher education Idiap Research Institute - IDIAP
Main discipline Information Technology
Start/End 01.07.2014 - 31.10.2018
Approved amount 237'423.00
Show all

Keywords (5)

manifold learning; generative modelling; heterogeneous face recognition; biometrics; session variability

Lay Summary (French)

L'une des tâches les plus difficiles en reconnaissance de visage est la mise en correspondance entre les images de visage acquises dans des environnements hétérogènes (spectres visuels, proche infrarouge, thermal, 3D, croquis).La difficulté principale est que les images d'un même sujet peuvent différer dans différents spectres introduisent de fortes variations intra-classe. La reconnaissance de visage hétérogène doit développer des représentations du visage invariantes à ces variations. Dans le projet HFACE, nous allons travailler sur plusieurs approches pour faire face à ces défis.
Lay summary

Comme première approche, nous allons étudier les techniques d'apprentissage de variétés (manifold). L'idée clé de cet apprentissage est de déterminer un sous-espace où les projections d'images de différentes sources peuvent être comparées directement grace à une fonction de distance.

Dans un deuxième temps, nous allons étudier les approches génératives pour la reconnaissance de visage. Dans ce contexte, la mise en correspondance hétérogène peux être considérée comme une variabilité inter-session qu'il est possible de modéliser et de compenser.

Finalement, une combinaison des deux approches sera étudiée. Cette combinaison peut être réalisée de deux manières différentes. Dans un premier temps , nous allons étudier la fusion des scores de reconnaissance de chaque approche. Nous allons ensuite étudier la possibilité d'utilser les projections de la première approche directement avec les modèles génératifs.

Direct link to Lay Summary Last update: 25.04.2014

Responsible applicant and co-applicants



Heterogeneous Face Recognition Using Domain Specific Units
Pereira Tiago de Freitas, Anjos Andre, Marcel Sebastien (2019), Heterogeneous Face Recognition Using Domain Specific Units, in IEEE Transactions on Information Forensics and Security, 1-13.
Cross-eyed 2017: Cross-spectral iris/periocular recognition competition
Sequeira Ana F., Chen Lulu, Ferryman James, Wild Peter, Alonso-Fernandez Fernando, Bigun Josef, Raja Kiran B., Raghavendra R., Busch Christoph, de Freitas Pereira Tiago, Marcel Sebastien, Behera Sushree Sangeeta, Gour Mahesh, Kanhangad Vivek (2017), Cross-eyed 2017: Cross-spectral iris/periocular recognition competition, in 2017 IEEE International Joint Conference on Biometrics (IJCB), Denver, COIEEE, 2017.
Heterogeneous Face Recognition Using Inter-Session Variability Modelling
Pereira Tiago de Freitas, Marcel Sebastien (2016), Heterogeneous Face Recognition Using Inter-Session Variability Modelling, in 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USAIEEE, 2016.
Periocular biometrics in mobile environment
de Freitas Pereira Tiago, Marcel Sebastien (2015), Periocular biometrics in mobile environment, in 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), Arlington, VA, USAIEEE, 2015.
Face liveness detection using dynamic texture
Freitas Pereira Tiago de, Komulainen Jukka, Anjos André, De Martino José Mario, Hadid Abdenour, Pietikäinen Matti, Marcel Sébastien (2014), Face liveness detection using dynamic texture, in EURASIP Journal on Image and Video Processing, 2014(1), 2-2.


Group / person Country
Types of collaboration
University of Campinas Brazil (South America)
- Publication
University of Colorado Colorado Springs United States of America (North America)
- Publication
University of Oulu Finland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication


Title Year
Idiap PhD student research 2018 2018

Use-inspired outputs


Face recognition has existed as a field of research for more than 30 years and has been particularly active since the early 1990s. Researchers of many different fields (from psychology, pattern recognition, neuroscience, computer graphics and computer vision) have attempted to create and understand face recognition systems.One of the most challenging tasks in automated face recognition is the matching between face images acquired in heterogeneous environments. Use-cases can cover matching of faces in unconstrained scenarios (e.g. at a distance), with long time lapse between the probe and the gallery and faces sensed in different modalities, such as thermal infrared or near infrared images (NIR) against visible spectra images (VIS). Successful solutions to heterogeneous face recognition can extend the reach of these systems to covert scenarios, such as recognition at a distance or at nighttime, or even in situations where no face even exists (forensic sketch recognition).The key difficult in matching faces from heterogeneous conditions is that images of the same subject may differ in appearance due to changes in image modality (e.g. between VIS images and NIR images, between VIS images and sketches images) introducing high intra-class variations. With these variations, a direct comparison between samples generally results in poor matching accuracy. Heterogeneous face recognition algorithms must develop facial representations invariant to these changes. In this proposal, we present three strategies to cope with these challenges.We will start by investigating manifold learning techniques. The key idea of manifold learning is to learn a joint mapping that project images, of different modalities, into a subspace where these projections can be compared directly with a simple distance function.As a second step, we will investigate generative approaches for face recognition. Generative approaches compute the likelihood of an observation (face image) or a set of observations given the a statistical model of the subject. In this environment, cross-sensor or hetereogeneous matching can be regarded as a session variability information the model needs to cope with, compensating for them during the enrollment and probing.Finally, a combination of both techniques will be investigated. This combination can be carried out in two different ways. At first, we will investigate score level fusion between the joint map learned with manifold learning approaches and the generative approach. We will then investigate the possibility to provide face images projected with the manifold learning techniques as input the generative face recognition systems.