Project

Back to overview

SONVB: Sensing and Analyzing Organizational Nonverbal Behavior

English title SONVB: Sensing and Analyzing Organizational Nonverbal Behavior
Applicant Gatica-Perez Daniel
Number 127542
Funding scheme Sinergia
Research institution IDIAP Institut de Recherche
Institution of higher education Idiap Research Institute - IDIAP
Main discipline Information Technology
Start/End 01.06.2010 - 31.12.2013
Approved amount 800'000.00
Show all

All Disciplines (2)

Discipline
Information Technology
Psychology

Keywords (8)

social interaction modeling; organizational behavior; nonverbal behavior; computational social science; social computing; social psychology; perceptual computing; ubiquitous computing

Lay Summary (English)

Lead
Lay summary
The understanding of organizational behavior has been a fundamental goal in social sciences in recent years. The development of theories and models to explain and improve key aspects of organizational life is increasingly relevant given the influence that people's jobs' scope and demands have on our lives. Face-to-face communication remains the primary form of social interaction in the workplace, and thus nonverbal communication plays a crucial role in organizational behavior. While spoken language constitutes a strong communication channel, a wealth of information is conveyed nonverbally in parallel to the spoken words, through tone of voice and prosody, and through proximity, body gestures and postures, gaze, and facial expressions. Nonverbal behavior (NVB) has different functions in social interactions. It not just serves to express emotions but also to signal attention, convey attitudes, and reveal personality characteristics. Furthermore, the establishment and evolution of multiple organizational constructs at the level of individuals and groups are the often measurable outcomes of face-to-face conversations with peers, superiors, and subordinates, where nonverbal communicative processes intervene through the display and interpretation of NVB.This interdisciplinary project proposes to advance the state-of-the-art in perceptual computing, ubiquitous computing, and social psychology for the analysis of NVB of individuals and groups in organizational scenarios. We aim to address three key interrelated aspects of behavior in organizations, namely leadership, personality, and performance. The project takes a broad view on the subject, through a carefully designed collaborative structure and common research objectives, in the context of dyadic and group interaction both in the laboratory and in real life. In social science, we aim to advance theories and empirical research on the interplay between first impressions, NVB, performance, and personality in job interviews and job execution; and between personality traits and exhibited NVB of members of small groups. In computer science, we aim to develop new computational methods for automatic extraction of NVB from face-to-face interaction from audio, video, and mobile sensors, and to design new computational models for recognition and discovery of social constructs. The project addresses a new interdisciplinary domain, capable of opening avenues for behavioral analysis research in psychology and computing, and provides a timely opportunity to train young scientists on the interplay between the human and computational facets of social interaction analysis.
Direct link to Lay Summary Last update: 21.02.2013

Responsible applicant and co-applicants

Employees

Publications

Publication
A semi-automated system for accurate gaze coding in natural dyadic interactions
Kenneth Funes, Laurent Nguyen, Daniel Gatica-Perez, Jean-Marc Odobez (2013), A semi-automated system for accurate gaze coding in natural dyadic interactions, in Proceedings of the 15th ACM on International conference on multimodal interaction, Sydney, Australia.
Body communicative cue extraction for conversational analysis
Marcos-Ramiro Alvaro, Pizarro-Perez Daniel, Marron-Romera Marta, Nguyen Laurent, Gatica-Perez Daniel (2013), Body communicative cue extraction for conversational analysis, in IEEE International Conference on Automatic Face and Gesture Recognition (FG), Shanghai.
Emergent leaders through looking and speaking: from audio-visual data to multimodal recognition
Sanchez-Cortes D., Aran O., Jayagopi D., Mast M. Schmid, Gatica-Perez D. (2013), Emergent leaders through looking and speaking: from audio-visual data to multimodal recognition, in Journal on Multimodal User Interfaces, in press, 7(1-2), 39-53.
Hiring gender-occupation incongruent applicants: The positive impact of recruiter interpersonal sensitivity
Frauendorfer Denise, Schmid Mast Marianne (2013), Hiring gender-occupation incongruent applicants: The positive impact of recruiter interpersonal sensitivity, in Journal of Personnel Psychology, 12, 182-188.
Inferring social activities with mobile sensor networks
Do Trinh-Minh-Tri, Kalimeri Kyriaki, Lepri Bruno, Pianesi Fabio, Gatica-Perez Daniel (2013), Inferring social activities with mobile sensor networks, in ACM International Conference on Multimodal Interaction (ICMI), Sydney.
Mining Large-Scale Smartphone Data for Personality Studies
Chittaranjan G., Blom J., Gatica-Perez D. (2013), Mining Large-Scale Smartphone Data for Personality Studies, in Personal and Ubiquitous Computing, 17(3), 433-450.
Multimodal analysis of body communication cues in employment interviews
Nguyen Laurent, Marcos-Ramiro Alvaro, Marron-Romera Marta, Gatica-Perez Daniel (2013), Multimodal analysis of body communication cues in employment interviews, in ACM International Conference on Multimodal Interaction (ICMI), Sydney.
One of a kind: Inferring personality impressions in meetings
Aran Oya, Gatica-Perez Daniel (2013), One of a kind: Inferring personality impressions in meetings, in ACM International Conference on Multimodal Interaction (ICMI), Sydney.
Personal informatics can be stressful: Collecting, reflecting, and embedding stress data in personal informatics
Voida Stephen, Choudhury Tanzeem, Gay Geri, Mathews Mark, Adams Phil, Rabbi Mashfiqui, Pollack JP, Chi Mengxi, Green Mathew, Lu Hong, Lane Nic, Mu Lin, Campbell Andrew (2013), Personal informatics can be stressful: Collecting, reflecting, and embedding stress data in personal informatics, in CHI Personal Informatics Workshop , Paris.
A Nonverbal Behavior Approach to Identify Emergent Leaders in Small Groups
Sanchez-Cortes D., Aran O., Schmid Mast M., Gatica-Perez D. (2012), A Nonverbal Behavior Approach to Identify Emergent Leaders in Small Groups, in IEEE Transactions on Multimedia, 14(3-2), 816-832.
Assessing the impact of language style on emergent leadership perception from ubiquitous audio
Sanchez-Cortes Dairazalia, Motlicek Petr, Gatica-Perez Daniel (2012), Assessing the impact of language style on emergent leadership perception from ubiquitous audio, in International Conference on Mobile and Ubiquitous Multimedia (MUM), Ulm.
Linking speaking and looking behavior patterns with group composition, perception, and performance
Jayagopi Dinesh, Sanchez-Cortes Dairazalia, Otsuka Kazuhiro, Yamato Junji, Gatica-Perez Daniel (2012), Linking speaking and looking behavior patterns with group composition, perception, and performance, in ACM International Conference on Multimodal Interaction (ICMI), Santa Monica.
StressSense: Detecting Stress in Unconstrained Acoustic Environments using Smartphones
Lu H., Rabbi M., Chittaranjan G., Frauendorfer D., Mast M., Campbell A., Gatica-Perez D., Choudhury T. (2012), StressSense: Detecting Stress in Unconstrained Acoustic Environments using Smartphones, in Proc. ACM Int. Conf. on Ubiquitous Computing, Pittsburgh.
Using self-context for multimodal detection of head nods in face-to-face interactions
Laurent Nguyen, Jean-Marc Odobez, Daniel Gatica-Perez (2012), Using self-context for multimodal detection of head nods in face-to-face interactions, in Proceedings of the 14th ACM international conference on Multimodal interaction, Santa Monica, USA.
An Audio Visual Corpus for Emergent Leader Analysis
Sanchez-Cortes D., Aran O., Gatica-Perez D. (2011), An Audio Visual Corpus for Emergent Leader Analysis, in ICMI Workshop on Multimodal Corpora for Machine Learning, 14(3-2), 14(3-2).
Analysis of Group Conversations: Modeling Social Verticality
Aran O., Gatica-Perez D. (2011), Analysis of Group Conversations: Modeling Social Verticality, Springer, London, 293-322.
Exploiting Observers' Judgments for Nonverbal Group Interaction Analysis
Chittaranjan Gokul, Aran Oya, Gatica-Perez Daniel (2011), Exploiting Observers' Judgments for Nonverbal Group Interaction Analysis, in IEEE Int. Conf. on Automatic Face and Gesture Recognition, Santa Barbara.
Self-promoting and modest job applicants in different cultures
Schmid Mast M., Frauendorfer D., Popovic L. (2011), Self-promoting and modest job applicants in different cultures, in Journal on Personnel Psychology, 10, 70-77.
Who is Who with Big-Five: Analyzing and Classifying Personality Traits with Smartphones
Chittaranjan G., Blom J., Gatica-Perez D. (2011), Who is Who with Big-Five: Analyzing and Classifying Personality Traits with Smartphones, in Proc. IEEE Int. Conf. on Wearable Computers, San Francisco.
Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior
Nguyen Laurent, Frauendorfer Denise, Schmid Mast Marianne, Gatica-Perez Daniel, Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior, in IEEE Transactions on Multimedia, 00.
Nonverbal social sensing in action: Unobtrusive recording and extracting of nonverbal behavior in social interactions illustrated with a research example
Frauendorfer Denise, Schmid Mast Marianne, Nguyen Laurent, Gatica-Perez Daniel, Nonverbal social sensing in action: Unobtrusive recording and extracting of nonverbal behavior in social interactions illustrated with a research example, in Journal of Nonverbal Behavior , 00.
The impact of nonverbal behavior in the job interview
Frauendorfer Denise, Schmid Mast Marianne, The impact of nonverbal behavior in the job interview, in Nonverbal Communication, 00.
Understanding Small Groups
Gatica-Perez Daniel, Aran Oya, Jayagopi Dinesh, Understanding Small Groups, in Magnenat-Thalmann Nadia, Burgoon Judee, Vinciarelli Alessandro, Pantic Maja (ed.), Cambridge University Press, UK, 00.
Where and what: Using smartphones to predict next locations and applications in daily life
Do Trinh-Minh-Tri, Gatica-Perez Daniel, Where and what: Using smartphones to predict next locations and applications in daily life, in Pervasive and Mobile Computing (PMC).

Collaboration

Group / person Country
Types of collaboration
University of Arizona United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Exchange of personnel
Nokia Research Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Research Infrastructure
- Industry/business/other use-inspired collaboration
Loyola Marymount University United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
University of Groningen Netherlands (Europe)
- in-depth/constructive exchanges on approaches, methods or results
Bruno Kessler Foundation Italy (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
Northeastern University United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
University of Alcala Spain (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
Dartmouth College United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Research Infrastructure

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
Talk, 13 Biannual Congress of the Swiss Psychological Society Individual talk 11.09.2013 Basel, Switzerland Frauendorfer Denise;
Talk, Symposium at the 13 Biannual Congress of the Swiss Psychological Society Individual talk 11.09.2013 Basel, Switzerland Schmid Mast Marianne;
Talk, 13 Biannual Congress of the Swiss Psychological Society Individual talk 11.09.2013 Basel, Switzerland Schmid Mast Marianne;
Talk, Association for Psychological Science Annual Conference, Smartphone Symposium Individual talk 23.05.2013 Washington DC, United States of America Choudhury Tanzeem;
Poster, 16th Congress of the European Association of Work and Organizational Psychology Poster 22.05.2013 Muenster, Germany Sanchez Cortes Dairazalia; Frauendorfer Denise; Gatica-Perez Daniel; Schmid Mast Marianne;
Invited Talk, Winter School on Context-Aware Systems Individual talk 18.02.2013 Klosters, Switzerland Gatica-Perez Daniel;
Poster, 14th Annual Meeting of the Society for Personality and Social Psychology Poster 17.01.2013 New Orleans, United States of America Frauendorfer Denise; Schmid Mast Marianne;
Keynote talk, Nonverbal Behavior Preconference of the 14th Annual Meeting of the Society for Personality and Social Psychology Individual talk 17.01.2013 New Orleans, United States of America Schmid Mast Marianne;
Keynote talk, ACM Workshop on Mobile Systems for Computational Social Science Individual talk 25.06.2012 Lake District, Great Britain and Northern Ireland Gatica-Perez Daniel;
Invited Talk, Summer School on Pattern Recognition in Multimodal Human Interaction Individual talk 04.06.2012 Vigo, Spain Gatica-Perez Daniel;
Invited talk, IEEE Signal Processing and Communications Applications Conference Individual talk 18.04.2012 Fethiye, Turkey Gatica-Perez Daniel;
Poster, 13th Conference of the Society for Social and Personality Psychology Poster 26.01.2012 San Diego, CA, USA, United States of America Frauendorfer Denise;
Invited talk, Portuguese Conference on Pattern Recogntion Individual talk 28.10.2011 Porto, Portugal Gatica-Perez Daniel;
Talk, NIH mHealth Evidence Workshop Individual talk 01.08.2011 Bethesda MD, USA, United States of America Choudhury Tanzeem;
Invited talk, 1st International Workshop on Frontiers in Activity Recognition using Pervasive Sensing 2011 Individual talk 12.06.2011 San Francisco, CA, USA, United States of America Choudhury Tanzeem;
Talk, NIH mHealth Summer Institute Individual talk 01.06.2011 San Diego, CA, USA, United States of America Choudhury Tanzeem;


Self-organised

Title Date Place
Ubicomp 2012 Tutorial: Building and Deploying Mobile Systems for Activity Recognition 17.09.2011 Beijing, China, China

Knowledge transfer events

Active participation

Title Type of contribution Date Place Persons involved
TEDx Martigny 14.09.2012 Martigny, Switzerland Gatica-Perez Daniel;
PopTech Conference 19.10.2011 Camden USA, United States of America Choudhury Tanzeem;


Communication with the public

Communication Title Media Place Year
Media relations: radio, television Il gesto rivelatore - Il Giardino di Albert RSI - Radio Television Switzerland in Italian Italian-speaking Switzerland 2013
Media relations: print media, online media Interview, Denise Frauendorfer Psychoscope, the journal of Fédération Suisse des Psychologues (FSP) German-speaking Switzerland Western Switzerland Italian-speaking Switzerland 2013
Media relations: radio, television Interview, Denise Frauendorfer Canal Alpha, TV Neuchatel Western Switzerland 2013
Media relations: print media, online media Microphones as sensors: Teaching old microphones new tricks The Economist International 2013
Media relations: print media, online media On the screen, gestures reach us better than text La Vanguardia, Spain International 2013
New media (web, blogs, podcasts, news feeds etc.) University Labs Driving Wearables Innovation Inside Activity Tracking: Tracking The Wearable Computing Revolution International 2013
Media relations: print media, online media Mindreader: Facebook of Revelations New Scientist International 2012
Media relations: print media, online media The Little Voice in Your Head NYTimes Magazine International 2012
Media relations: print media, online media When the body talks SNSF Horizons Magazine Western Switzerland Italian-speaking Switzerland German-speaking Switzerland 2012
New media (web, blogs, podcasts, news feeds etc.) An App That Reads Your Feelings Through Your Voice. Fast Company's Co.Exist International 2011

Awards

Title Year
Grant for the one month visit at University of Arizona, Equal Opportunity Committee University of Neuchatel 2013
Outstanding Paper Award, ACM ICMI 2012 2012
Nomination for best paper award, IEEE ISWC 2011 2011

Associated projects

Number Title Start Funding scheme
147611 UBImpressed: Ubiquitous First Impressions and Ubiquitous Awareness 01.01.2014 Sinergia
136811 Multimodal Computational Modeling of Nonverbal Social Behavior in Face to Face Interaction 01.12.2011 Ambizione
147611 UBImpressed: Ubiquitous First Impressions and Ubiquitous Awareness 01.01.2014 Sinergia

Abstract

The understanding of organizational behavior has been a fundamental goal in social sciences in recent years. The development of theories and models to explain and improve key aspects of organizational life has an undeniable value given the increasing influence that people’s jobs’ scope and demands, their associated styles and rituals, and the resulting professional and personal relationships have on their lives. Face-to-face communication remains the primary form of social interaction in the workplace, and thus nonverbal communication plays a crucial role in organizational behavior. While spoken language constitutes a strong communication channel, a wealth of information is conveyed nonverbally in parallel to the spoken words, through tone of voice and prosody, and through proximity, body gestures and postures, gaze, and facial expressions. Nonverbal behavior (NVB) has different functions in social interactions. It not just serves to express emotions but also to signal attention, convey attitudes about friendliness or dominance, and reveal personality characteristics such as shyness or extraversion. Furthermore, the establishment and evolution of multiple organizational constructs at the level of individuals - including judgments of personality, status, and competence - and groups - the emergence of leadership, cohesion, and performance - are the often measurable outcomes of the myriad of micro-level face-to-face conversations with peers, superiors, and subordinates, where nonverbal communicative processes intervene through the (conscious or unconscious) display and interpretation of NVB.This interdisciplinary research program proposes to significantly advance the state-of-the-art in perceptual computing, ubiquitous computing, and social psychology for the computational sensing and analysis of NVB of individuals and groups in organizational scenarios at large-scale. More specifically, we aim to address three key interrelated aspects of behavior in organizations, namely leadership, personality, and performance. These behavioral constructs are not only ubiquitous at work but also key in the formation and maintenance of relationships, and ultimately in the success of individuals and teams. This project takes a novel and broad view on the subject. Through a carefully designed collaborative structure and common research objectives, we will study the relations between short-term and long-term NVB and the emergence of leadership, the assessment of personality, and the prediction of job performance, as well as the interplay among these constructs, by analyzing dyadic and small group interaction both in the laboratory and in real-life, using automatic methods and involving multiple sensors (cameras, microphones, and wearable devices). In social science we aim at advancing theories and empirical research on the interplay between first impressions, NVB, performance, and personality in job interviews and job execution; and between personality traits and exhibited NVB of emergent leaders in small groups over long-term interaction. In computer science, we aim to develop new computational methods for automatic extraction of NVB from face-to-face interaction from audio, video, and mobile sensors, and to design new computational models for recognition and discovery of social constructs from brief and long-term observations of nonverbal behavior, including leadership in small groups, personality traits in the workplace, and job performance. The proposed research program addresses a new interdisciplinary domain, which has been recognized as disruptive and capable of opening entirely new avenues for behavioral analysis research in psychology and perceptual and ubiquitous computing, and provides a timely opportunity to train young scientists on the interplay between the human and computational facets of social interaction analysis.
-