Back to overview

Storytelling and first impressions in face-to-face and algorithm-powered digital interviews

English title Storytelling and first impressions in face-to-face and algorithm-powered digital interviews
Applicant Bangerter Adrian
Number 197479
Funding scheme Project funding
Research institution IPTO - Institut de Psychologie du Travail et des Organisations Université de Neuchâtel
Institution of higher education University of Neuchatel - NE
Main discipline Psychology
Start/End 01.02.2021 - 31.01.2025
Approved amount 700'325.00
Show all

All Disciplines (2)

Information Technology

Keywords (7)

social computing; Selection interview; validity; machine learning; storytelling; non-verbal behavior; verbal behavior

Lay Summary (French)

L’entretien d’embauche comportemental permet de prédire la performance future des candidat-e-s à un poste, en leur demandant de raconter des situations de travail qu’ils ont géré dans le passé, et en analysant les réponses qu’ils donnent. A l’heure actuelle, on connaît mal comment les candidat-e-s se comportent dans de telles situations. De plus en plus, les entretiens deviennent digitaux: Le comportement est analysé au moyen d’algorithmes.
Lay summary
Ce projet est une collaboration interdisciplinaire entre des psychologues et des informaticiens, avec trois objectifs : (1) Comprendre comment la performance de narration reflète la maîtrise d'une compétence spécifique par rapport à des capacités plus génériques (charisme), (2) comprendre les processus derrière les différences de performance dans les entretiens digitauxpar rapport aux entretiens en face à face, et (3) développer une méthode pour l'analyse automatique des discours produits dans les entretiens digitaux à l'aide de l'intelligence artificielle, en améliorant la reconnaissance automatique de la parole et l'analyse de contenu. 
Direct link to Lay Summary Last update: 22.12.2020

Responsible applicant and co-applicants


Associated projects

Number Title Start Funding scheme
185010 NAST: Neural Architectures for Speech Technology 01.02.2020 Project funding
184858 Behavioral Adaptability 01.09.2020 Project funding
183564 Go Where No-One Has Gone: Virtual Reality for Interpersonal Skills Training 01.06.2019 Sinergia
183065 Automatic analysis of verbal and non-verbal behavior and provision of feedback in video selection interviews 01.12.2018 Digital Lives


The structured selection interview is a valid predictor of job performance, often featuring so-called past-behavior questions, which invite applicants to tell a story about a past work situation. Applicants’ storytelling responses are often suboptimal. Moreover, it is unclear what they reflect: Mastery of a specific competency, or a more general trait (e.g., charisma). Recently, asynchronous digital interviews have emerged, where applicant behaviour (typically nonverbal) is videorecorded and automatically analysed via machine learning algorithms. Applicants perform worse in these interviews than face-to-face. The cause of this difference is poorly understood; moreover, the criterion validity of digital interviews for predicting job performance is unknown. Analysis of verbal behavior is less advanced, being limited by automatic speech recognition (ASR) and standards for defining content for natural language processing (NLP) analyses. Storytelling in response to past-behavior questions provides a yardstick for “good” content, and could be identified by building state-of-the-art ASR and NLP approaches allowing integration of verbal and nonverbal behavior.This project is an interdisciplinary collaboration between psychologists and computer scientists with three goals: (1) Understanding how storytelling performance reflects the mastery of a specific competency versus more generic abilities (charisma), (2) understanding processes behind performance differences in technology-mediated versus face-to-face interviews, and (3) improving the verbal pipeline for AI-powered digital interviews by improving ASR and NLP. We start by analyzing existing data on how applicants respond to past-behavior questions in digital versus face-to-face interviews (Studies 1 and 2). In the main study (Study 3), participants complete a work sample to measure task performance, and later are asked to tell what they did in a simulated interview, in either a face-to-face or a digital condition. Participants’ behavior in the interview and work sample will be transcribed and coded to compute criterion validity. Responses in face-to-face and digital conditions will be compared. The data will be used to build an ASR and to extract storytelling using end-to-end machine learning. The planned research will make key contributions to personnel selection research and computer science, in terms of scientific publications, software development and training and education.