Project

Back to overview

Human activity and interactivity modeling (HAI)

English title Human activity and interactivity modeling (HAI)
Applicant Odobez Jean-Marc
Number 132620
Funding scheme Project funding (Div. I-III)
Research institution IDIAP Institut de Recherche
Institution of higher education Idiap Research Institute - IDIAP
Main discipline Information Technology
Start/End 01.10.2010 - 31.03.2014
Approved amount 218'856.00
Show all

Keywords (12)

Activity analysis; social behavior; interactions; video; cell-phone; social media; language processing; data mining; topic models; sequence models; context modeling; large-scale sensing

Lay Summary (English)

Lead
Lay summary
Understanding human behavior is one of the most intriguing research domains, which encompasses research fields ranging from sociology to computer science. Immense progress in sensor and communication technologies has led to the development of devices and systems recording daily human activities in both real and virtual (web-based) settings. This has led to an increase of research on algorithms capable of inferring meaningful behavioral patterns of human activities from the information contained in data logs or captured by sensors. Simultaneously, there are new application opportunities such as surveillance, health care monitoring, social networking, and recommendation systems.The HAI project will investigate the above domain by performing long-term research addressing fundamental questions and common tasks: how to design robust features for accurate activity/interaction representation? How to learn or discover activity patterns, introduce hierarchies or temporal order at different scales? how to infer contextual information that affects activity patterns or their occurrence? To this end, we will investigate new approaches by anchoring the design of general activity models relying on principled methodologies (bag-of-words and Bayesian topic models) in the context of three different application domains with specific scenarios, types of activities, and data modalities.HAI-1: Activity analysis from long term video recordings. The goal is to automatically discover the typical activities of moving entities (cars, people, groups), their characteristics, the relations between them, and detect abnormal activities, by combining sequential state models at different levels with data mining tools relying on co-occurence analysis.HAI-2: Activity analysis from mobile phone data. The aim is the design of novel heterogenous data representations and probabilistic models for the modeling of varying time duration routines, the identification of large scale human communication patterns, and the discovery of individual's life patterns from a rich set of phone data modalities (location, proximity, phone calls, text messaging). HAI-3: Community activity analysis in social media. The goal is to investigate the structure, evolution, and practices of communities in social media with Flick as a target. In particular, using statistical models relying on textual and social metadata, the project will model the dynamical aspect of social media groups (including topic and memberships patterns), study and discover micro-activity patterns within sub-groups, and investigate the use of visual information extracted from photos and videos to refine user and group descriptions.
Direct link to Lay Summary Last update: 21.02.2013

Responsible applicant and co-applicants

Employees

Publications

Publication
A probabilistic approach to mining mobile phone data sequences
Farrahi Katayoun, Gatica-Perez Daniel (2014), A probabilistic approach to mining mobile phone data sequences, in Personal and Ubiquitous Computing, 18(1), 223-238.
From Foursquare to My Square: Learning Check-in Behavior from Multiple Sources
Malmi Eric, Do Trinh Minh Tri, Gatica-Perez Daniel (2013), From Foursquare to My Square: Learning Check-in Behavior from Multiple Sources, in ICWSM.
Revisiting the generality of the rank-based human mobility model
Santani Darshan, Gatica-Perez Daniel (2013), Revisiting the generality of the rank-based human mobility model, in Ubicomp (Adjunct Publication).
Speaking swiss: languages and venues in foursquare
Santani Darshan, Gatica-Perez Daniel (2013), Speaking swiss: languages and venues in foursquare, in ACM Multimedia, Barcelone.
Bridging the past, present and future: Modeling scene activities from event relationships and global rules
Jagan Varadarajan, Rémi Emonet, Jean-Marc Odobez, Farrahi K., Pentland A. (2012), Bridging the past, present and future: Modeling scene activities from event relationships and global rules, in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2012, Providence.
Checking In or Checked In: Comparing Large-Scale Manual and Automatic Location Disclosure Patterns
Malmi E., Do T., Gatica-Perez D. (2012), Checking In or Checked In: Comparing Large-Scale Manual and Automatic Location Disclosure Patterns, in Proc. Int. Conf. on Mobile and Ubiquitous Multimedia (MUM), Ulm, Oral Presentation, Ulm.
Extracting Mobile Behavioral Patterns with the Distant N-Gram Topic Model
Farrahi K., Gatica-Perez D. (2012), Extracting Mobile Behavioral Patterns with the Distant N-Gram Topic Model, in Proc. IEEE Int. Symp. on Wearable Computers (ISWC), Newcastle.
Flickr groups: Multimedia communities for multimedia analysis
Negoescu R., Gatica-Perez D., Odobez J-M (2012), Flickr groups: Multimedia communities for multimedia analysis, Bentham Science Publishers, Unknown, 234-250.
A Probabilistic Approach to Socio-Geographic Reality Mining
Farrahi K., Emonet R., Odobez J.-M. (2011), A Probabilistic Approach to Socio-Geographic Reality Mining, PhD Thesis, EPFL, Lausanne.
Discovering Routines from Large-Scale Human Locations using Probabilistic Topic Models
Farrahi K., Gatica-Perez D., Gatica-Perez D. (2011), Discovering Routines from Large-Scale Human Locations using Probabilistic Topic Models, in ACM Transactions on Intelligent Systems and Technology, Special Issue on Activity Recognition, 2(1), 137-162.
Extracting and locating temporal motifs in video scenes using a hierarchical non parametric Bayesian model
Emonet R, Varadarajan J, Odobez J-M (2011), Extracting and locating temporal motifs in video scenes using a hierarchical non parametric Bayesian model, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, Providence, Rhode IslandProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Colorado Springs.
Modeling and Understanding Communities in Online Social Media using Probabilistic Methods
Negoescu R. (2011), Modeling and Understanding Communities in Online Social Media using Probabilistic Methods, PhD Thesis, EPFL, Lausanne.
Multi-camera open space human activity discovery for anomaly detection
Emonet R, Varadarajan J (2011), Multi-camera open space human activity discovery for anomaly detection, in 8th IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS), 2011 , Klagenfurth, Austria2011 8th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2011, Klagenfuhrt, Austria.
Pervasive Sensing to Model Political Opinions in Face-to-Face Networks
Madan A. (2011), Pervasive Sensing to Model Political Opinions in Face-to-Face Networks, in Proc. Int. Conf. on Pervasive Computing (Pervasive), San FransiscoInt. Conf. on Pervasive Computing (Pervasive), San-Francisco.
Sensing the Health State of our Society
Madan A., Cebrian M., Moturu S., Farrahi K., Pentland A. (2011), Sensing the Health State of our Society, in IEEE Pervasive Computing, 1-15.
A Sparsity Constraint for Topic Models: Application to Temporal Activity Mining.
Varadarajan J., Emonet R. (2010), A Sparsity Constraint for Topic Models: Application to Temporal Activity Mining., in NIPS Workshop on Workshop on Practical Applications of Sparse Modeling: Open Issues and New Directio, Vancouver, CanadaNIPS workshop on Practical Application of Sparse Modeling: Open Issues and New Directions, Vancouver, Canada.
Kodak Moments and Flickr Diamonds: How Users Shape Large-Scale Media
Negoescu R., Loui A., Gatica-Perez D., Pentland A. (2010), Kodak Moments and Flickr Diamonds: How Users Shape Large-Scale Media, in Proc. ACM Int. Conf. on Multimedia (ACM MM), , Florence, ItalieACM Int. Conf. on Multimedia (ACM MM), Firenze.
Mining Human Location-Routines Using a Multi-Level Approach to Topic Modeling
Farrahi Katayoun, Gatica-Perez daniel (2010), Mining Human Location-Routines Using a Multi-Level Approach to Topic Modeling, in IEEE Int. Conference on Social Computing, Symposium on Social Intelligence and Networking (SocialCom, Proceedings of the IEEE International Conference on Social Computing, SIN Symposium, Minneapolis, USA.
Modeling and Understanding Flickr Communities through Topic-based Analysis
Negoescu Radu-Andrei, Gatica-Perez Daniel (2010), Modeling and Understanding Flickr Communities through Topic-based Analysis, in IEEE Transactions on Multimedia, 12(5), 399-416.
Probabilistic Latent Sequential Motifs: Discovering Temporal Activity Patterns in Video Scenes
Varadarajan J., Emonet R., Odobez J.-M. (2010), Probabilistic Latent Sequential Motifs: Discovering Temporal Activity Patterns in Video Scenes, in Proc. British Machine and Vision Conference (BMVC), Sept. 2010, AberystwythBritish Machine Vision Conference (BMVC), Aberystwyth, UK.
A sequential topic model for mining recurrent activities from long term video logs
Jagan Varadarajan, Rémi Emonet, Jean-Marc Odobez, A sequential topic model for mining recurrent activities from long term video logs, in Int. Journal of Computer Vision (IJCV).

Collaboration

Group / person Country
Types of collaboration
Human Dynamics Lab at MIT Media Lab United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
Doctoral Consortium Poster Mining activities: from events to relationship 25.06.2012 Providence, Rhode Island, United States of America Varadarajan Jagannadan;
Ecole d'été Poster Unsupervised mining of activity from large data logs 04.09.2011 Bordeaux, France, France Varadarajan Jagannadan;


Self-organised

Title Date Place
Human Activity and Vision Summer School 01.10.2012 Antibes, France, France

Communication with the public

Communication Title Media Place Year
Media relations: print media, online media Surveying African Cities using Twitter International 2014
Media relations: print media, online media Social Media: Schweiz schreibt Englisch Computerworld.ch German-speaking Switzerland 2013
Talks/events/exhibitions Analyses automatiques des vidéos : principes et perspectives Western Switzerland 2010

Awards

Title Year
Nomination for best paper award, ISWC conference 2012
Google Anita Borg Memorial Scholarship 2010
Idiap PhD Student Research award 2010

Associated projects

Number Title Start Funding scheme
122062 MULTI: Multimodal Interaction and Multimedia Data Mining 01.10.2008 Project funding (Div. I-III)

Abstract

Understanding human behavior is one of the most intriguing and fascinating research domains, which encompasses several research fields, ranging from economics and sociology to more recently computer science. Immense progress in sensor and communication technologies has led to the development of devices and systems recording daily human activities in both real and virtual (web-based) settings. This has led to an increase of research on the design of algorithms capable of inferring meaningful behavioral patterns of human activities from the information contained in data logs or captured by sensors. Simultaneously, there are new application opportunities in many domains such as surveillance, health care monitoring, social networking, and recommendation systems.The aim of the HAI project is to investigate the above domain by performing long-term research which addresses fundamental questions and common tasks of this domain: how to design robust features for accurate activity/interaction representation? How to learn or discover activity patterns, introduce hierarchies or temporal order at different scales, and deal efficiently with large amounts of data? How to infer contextual information that affects activity patterns or their occurrence, or facilitates their interpretation. To achieve these goals, we will investigate new approaches by anchoring the design of general activity models in the context of four different and relatively recent application domains with specific scenarios, types of activities, and data modalities.HAI-1: Activity analysis from long term video recordings. The goal is to automatically discover the typical activities of moving entities (cars, people, groups), their characteristics, the relations between them within and across cameras, and detect abnormal activities. These goals will be reached by combining sequential state models at different levels with data mining tools relying on co-occurrence analysis.HAI-2: Activity analysis from mobile phone data. The aim is the design of novel heterogenous data representations and probabilistic models for the modeling of varying time duration routines for location and proximity based activity discovery, for the identification of large scale human communication patterns based on phone calls or text messaging, and for the discovery of individual's life patterns from a rich set of phone data modalities.HAI-3: Community activity analysis in social media. The goal is to investigate the structure, evolution, and practices of communities in social media with Flick as a target. In particular, using statistical models relying on textual and social metadata, the project will model the dynamical aspect of social media groups (including topic and memberships patterns), study and discover micro-activity patterns within sub-groups, and investigate the use of visual information extracted from photos and videos to refine user and group descriptions.HAI-4: Context modeling for just-in-time multimedia information retrieval. The goal is to model the context of conversational activity for users of an Automatic Content Linking Device (ACLD), which is a just-in-time retrieval system that spontaneously displays documents and multimedia fragments based on the current topic of discussion. Context modeling will help to decide whether it is useful and appropriate to interrupt users with new results, and to estimate relevance feedback from users in order to guide upcoming searches.While each of the sub-projects pursues its own goals, the grounding of the approaches on similar principled methodologies (e.g. bag-of-words and Bayesian topic models) will provide opportunities for research synergies and the strengthening of Idiap's activities on human behavior and interaction modeling.
-