Project

Back to overview

Automatic analysis of verbal and non-verbal behavior and provision of feedback in video selection interviews

English title Automatic analysis of verbal and non-verbal behavior and provision of feedback in video selection interviews
Applicant Bangerter Adrian
Number 183065
Funding scheme Digital Lives
Research institution IPTO - Institut de Psychologie du Travail et des Organisations Université de Neuchâtel
Institution of higher education University of Neuchatel - NE
Main discipline Psychology
Start/End 01.12.2018 - 30.11.2020
Approved amount 250'000.00
Show all

All Disciplines (2)

Discipline
Psychology
Information Technology

Keywords (7)

feedback; machine learning; social computing; Selection interview; social sensing; non-verbal behavior; Verbal behavior

Lay Summary (French)

Lead
L’entretien vidéo est une procédure de sélection où les candidat-e-s à un emploi s’enregistrent en vidéo en train de répondre à des questions qui leur sont préalablement envoyés, pour ensuite envoyer la vidéo à l’employeur potentiel. De plus en plus utilisés, les entretiens vidéo sont susceptibles d’être analysés automatiquement au moyen d’algorithmes. Ce projet étudie la manière dont les candidat-e-s potentiels à l’embauche se présentent sur des entretiens vidéo. Le projet développera des algorithmes pour analyser de manière valide le comportement verbal et non-verbal des candidat-e-s afin de leur fournir du feedback sur leur performance.
Lay summary

L’entretien vidéo est une procédure de sélection où les candidat-e-s à un emploi s’enregistrent en vidéo en train de répondre à des questions qui leur sont préalablement envoyés, pour ensuite envoyer la vidéo à l’employeur potentiel. Les entretiens vidéo sont de plus en plus utilisés car le comportement du candidat peut être analysé automatiquement au moyen d’algorithmes. Cependant, il n’est pas clair en fonction de quoi les algorithmes sont calibrés, autrement dit si les sélections opérées sont effectuées sur des bases valides ou pas. Ce projet étudie la manière dont les candidat-e-s potentiels à l’embauche se présentent sur des entretiens vidéo. Le projet développera des algorithmes pour analyser de manière valide le comportement verbal et non-verbal des candidat-e-s afin de leur fournir du feedback sur leur performance.

L’utilisation d’algorithmes permet de fournir des informations très détaillées aux candidat-e-s sur leur comportement. Cependant, il n’est pas clair si les candidat-e-s peuvent utiliser cette information pour améliorer leurs performances. Le projet testera différentes manières de founrir du feedback quant à leur utilité pour les candidat-e-s.

Direct link to Lay Summary Last update: 01.10.2018

Responsible applicant and co-applicants

Employees

Publications

Publication
Understanding Applicants' Reactions to Asynchronous Video Interviews Through Self-reports and Nonverbal Cues
Muralidhar Skanda, Kleinlogel Emmanuelle Patricia, Mayor Eric, Bangerter Adrian, Schmid Mast Marianne, Gatica-Perez Daniel (2020), Understanding Applicants' Reactions to Asynchronous Video Interviews Through Self-reports and Nonverbal Cues, in ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, Virtual Event NetherlandsACM Proceedings, New York.

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
2020 ACM International Conference on Multimodal Interaction Talk given at a conference Understanding Applicants' Reactions to Asynchronous Video Interviews Through Self-reports and Nonverbal Cues 25.10.2020 Utrecht, Netherlands Muralidhar Skanda; Bangerter Adrian; Mayor Eric; Schmid Mast Marianne; Gatica-Perez Daniel; Kleinlogel Emmanuelle;
Applied Machine Learning Days Talk given at a conference Conventional versus non-conventional methods for job interviews 25.01.2020 Lausanne, Switzerland Kleinlogel Emmanuelle;
Applied Machine Learning Days Talk given at a conference Automatic extraction of storytelling in job interviews 25.01.2020 Lausanne, Switzerland Bangerter Adrian; Mayor Eric;
Applied Machine Learning Days Talk given at a conference Automatic analysis of videotaped job applicants: Promises and limits 25.01.2020 Lausanne, Switzerland Schmid Mast Marianne;
Applied Machine Learning Days Talk given at a conference Facing Employers and Customers: Cues for Perceived Soft Skills 25.01.2020 Lausanne, Switzerland Muralidhar Skanda;


Communication with the public

Communication Title Media Place Year
Media relations: radio, television Algorithmes et entretien embauche RTS Western Switzerland 2019
Media relations: print media, online media Analyser l’entretien d’embauche par l’intelligence artificielle? HR Today Western Switzerland 2019
Media relations: print media, online media Arriva il reclutatore virtuale ai colloqui di lavoro La Stampa International 2019
Media relations: radio, television Ne plus se déplacer pour un entretien d'embauche canal alpha Western Switzerland 2019

Associated projects

Number Title Start Funding scheme
152920 Storytelling in the selection interview: Antecedents, process and outcomes 01.12.2014 Project funding
197479 Storytelling and first impressions in face-to-face and algorithm-powered digital interviews 01.02.2021 Project funding

Abstract

This project investigates social interaction in personnel selection interviews enhanced by digital technology. We will create a database of applicants participating in video interviews (applicants receive a list of interview questions from a recruiter online and then record themselves answering those questions), which are a newly emerging interview format. We will develop automated procedures for extracting relevant behavioral features from streams of applicants’ verbal and nonverbal behavior in these interviews. This information will be (1) linked to external criteria (e.g., hireability ratings by expert recruiters), (2) used to train machine learning algorithms, and (3) fed back to the applicants. We will assess applicants’ perceptions of this feedback, whether and how they use it to improve their performance in a second video interview a day later, and how they perceive data privacy issues related to the use of their data.The project addresses three issues mentioned in the call. First, how is digitalization transforming social ties? The selection interview is the gateway to employment and thus the potential beginning of one of the fundamental social ties in modernity: the work relationship. We explore a new format by which selection interviews are conducted in an online, asynchronous manner. Second, how is digitalization transforming the economy? The selection interview is an important personnel selection procedure, which itself is an important component of strategic talent management. The digitalization of talent management is rapidly expanding in practice, but is currently poorly understood in research. Third, how is digitalization transforming our subjective experience? Video interviews are a novel experience for many applicants. Machine learning techniques can be used to extract the applicants’ behaviors recorded on the videos and to some degree infer their personality and social skills. This information can then be fed back to the applicants, potentially changing their subjective experience of the video interview. However, questions like how such feedback is best provided and how the applicant apprehends and uses it are largely unexplored.The study will yield four main sets of outputs. First, the primary data from the study will lead to publications in scientific journals or conference proceedings in human-computer interaction and organizational psychology or human resources. Second, the data will be used to adapt an existing data collection platform and to improve the quality of algorithms to infer verbal and nonverbal behavior from videos. Third, data about user experiences will inform the development of evidence-based coaching programs for improving applicants’ performance. Fourth, the rich set of data and experience generated will constitute fruitful avenues for further research by the applicant team.
-