Projekt

Zurück zur Übersicht

IMOTION - Intelligent Multi-Modal Augmented Video Motion Retrieval System

Titel Englisch IMOTION - Intelligent Multi-Modal Augmented Video Motion Retrieval System
Gesuchsteller/in Schuldt Heiko
Nummer 151571
Förderungsinstrument ERA-NET
Forschungseinrichtung Fachbereich Informatik Departement Mathematik und Informatik Universität Basel
Hochschule Universität Basel – BS
Hauptdisziplin Informatik
Beginn/Ende 01.01.2014 - 31.12.2017
Bewilligter Betrag 685'026.00
Alle Daten anzeigen

Keywords (6)

speech-based user interfaces; sketch-based user interfaces; video motion descriptors; index structures; motion queries; video retrieval

Lay Summary (Deutsch)

Lead
Ziel des IMOTION-Projekts ist es, neuartige multimodale Benutzerschnittstellen zu entwickeln und evaluieren, die es erlauben, Videos bzw. Ausschnitte aus Videos auf der Basis von Skizzen bzw. gesprochenen Anfragen zu suchen.
Lay summary

Ziel des IMOTION-Projekts ist es, neuartige multimodale Benutzerschnittstellen zu entwickeln und evaluieren, die es erlauben, Videos bzw. Ausschnitte aus Videos auf der Basis von Skizzen bzw. gesprochenen Anfragen zu suchen. Dies geschieht mit Hilfe von räumlichen Metadaten, die die Bewegung von Objekten in den Videos über mehrere Frames hinweg beschreiben. Die neuartigen skizzen- und sprachbasierten Anfrageparadigmen werden kombiniert mit bestehenden Verfahren wie z.B. der Schlagwortsuche in textuellen Annotationen oder der Bildähnlichkeitssuche (angewandt auf einzelne Video-Frames). Das Projekt wird mehrere innovative Benutzerschnittstellen entwickeln (für Sprache, Multi-Touch, Gesten, Skizzen, z.B. auf Tablets oder interaktivem Papier). Diese Schnittstellen sollen nahtlos integriert werden, so dass die Migration von Benutzersitzungen zwischen verschiedenen Benutzerschnittstellen ermöglicht wird um eine Suche zu verfeinern (z.B. startet ein Nutzer eine Schlagwortsuche, verfeinert diese mit einer Skizze, die eine Bewegungsgeste enthält und gibt schliesslich weitere Details in gesprochener Sprache an).

Direktlink auf Lay Summary Letzte Aktualisierung: 05.12.2013

Lay Summary (Englisch)

Lead
The IMOTION project will develop and evaluate innovative multi-modal user interfaces forinteracting with augmented videos (i.e., videos enriched with spatio-temporal metadata on the movement of objects). IMOTION will provide novel sketch- and speech-based user interfaces. In particular, novel types of motion queries will be supported where users can specify motion paths of objects, via sketches, gestures, natural language interfaces, or combinations thereof.
Lay summary
The IMOTION project will develop and evaluate innovative multi-modal user interfaces for interacting with augmented videos (i.e., videos enriched with spatio-temporal metadata on the movement of objects). Starting with an extension of existing query paradigms (keyword search in manual annotations), image search (query by example in key frames), IMOTION will provide novel sketch- and speech-based user interfaces. In particular, novel types of motion queries will be supported where users can specify motion paths of objects, via sketches, gestures, natural language interfaces, or combinations thereof. Several types of user interfaces (voice, tablets, multi-touch tables, interactive paper) will be supported and seamlessly combined so as to smoothly migrate a session from one type of user interface to another during the process of specifying and refining a query. This will be based on novel approaches to representation learning and the extraction of high-level motion descriptors from augmented videos, based on a motion ontology. In addition, IMOTION will develop novel index structures that jointly support traditional video features and the additional motion metadata.
Direktlink auf Lay Summary Letzte Aktualisierung: 05.12.2013

Verantw. Gesuchsteller/in und weitere Gesuchstellende

Mitarbeitende

Projektpartner

Publikationen

Publikation
Enhanced Retrieval and Browsing in the IMOTION System
Rossetto Luca, Giangreco Ivan, Tănase Claudiu, Schuldt Heiko, Dupont Stéphane, Seddati Omar (2017), Enhanced Retrieval and Browsing in the IMOTION System, in Proceedings of the 23rd International Conference on Multimedia Modeling, Reykjavik, IcelandSpringer, Heidelberg, Germany.
ADAMpro: Database Support for Big Multimedia Retrieval
Giangreco Ivan, Schuldt Heiko (2016), ADAMpro: Database Support for Big Multimedia Retrieval, in Datenbank-Spektrum, 16(1), 17-26.
Dealing with ambiguous Queries in Multimodal Video Retrieval
Rossetto Luca, Tănase Claudiu, Schuldt Heiko (2016), Dealing with ambiguous Queries in Multimodal Video Retrieval, in Proceedings of the 22nd International Conference on Multimedia Modeling (MMM 2016), Miami, FL, USASpringer, Heidelberg, Germany.
iAutoMotion ‐ an Autonomous Content‐based Video Retrieval Engine
Rossetto Luca, Giangreco Ivan, Tănase Claudiu, Schuldt Heiko, Dupont Stéphane, Seddati Omar, Sezgin Metin, Sahillioğlu Yusuf (2016), iAutoMotion ‐ an Autonomous Content‐based Video Retrieval Engine, in Proceedings of the 22nd International Conference on Multimedia Modeling (MMM 2016), Miami, FL, USASpringer, Heidelberg, Germany.
IMOTION ‐ Searching for Video Sequences using Multi‐Shot Sketch Queries
Rossetto Luca, Giangreco Ivan, Heller Silvan, Tănase Claudiu, Schuldt Heiko, Dupont Stéphane, Seddati Omar, Sezgin Metin, Altıok Ozan Can, Sahillioğlu Yusuf (2016), IMOTION ‐ Searching for Video Sequences using Multi‐Shot Sketch Queries, in Proceedings of the 22nd International Conference on Multimedia Modeling (MMM 2016), Miami, FL, USASpringer, Heidelberg, Germany.
Interactive video search tools: a detailed analysis of the video browser showdown 2015
Cobârzan Claudiu, Schoeffmann Klaus, Bailer Werner, Hürst Wolfgang, Blažek Adam, Lokoč Jakub, Vrochidis Stefanos, Barthel Kai Uwe, Rossetto Luca (2016), Interactive video search tools: a detailed analysis of the video browser showdown 2015, in Multimedia Tools and Applications, 1-33.
Searching in Video Collections using Sketches and Sample Images ‐ The Cineast System
Rossetto Luca, Giangreco Ivan, Heller Silvan, Tănase Claudiu, Schuldt Heiko (2016), Searching in Video Collections using Sketches and Sample Images ‐ The Cineast System, in Proceedings of the 22nd International Conference on Multimedia Modeling (MMM 2016), Miami, FL, USASpringer, Heidelberg, Germany.
Semantic Sketch‐Based Video Retrieval with Autocompletion
Tănase Claudiu, Giangreco Ivan, Rossetto Luca, Schuldt Heiko, Seddati Omar, Dupont Stéphane, Altiok Ozan Can, Sezgin Metin (2016), Semantic Sketch‐Based Video Retrieval with Autocompletion, in Proceedings of the 21st ACM International Conference on Intelligent User Interfaces (IUI'16), Sonoma, CA, USAACM, New York, NY, USA.
The IMOTION System at TRECVID 2016: The Ad-Hoc Video Search Task
Tanase Claudiu, Rossetto Luca, Giangreco Ivan, Schuldt Heiko, Dupont Stéphane, Seddati Omar (2016), The IMOTION System at TRECVID 2016: The Ad-Hoc Video Search Task, in Proceedings of the 2016 TRECVID Ad-Hoc Video Search Task, Gaithersburg, MD, USANIST, Gaithersburg, MD, USA.
The vitrivr System at TRECVID 2016: The Ad-Hoc Video Search Task
Tanase Claudiu, Rossetto Luca, Giangreco Ivan, Schuldt Heiko (2016), The vitrivr System at TRECVID 2016: The Ad-Hoc Video Search Task, in Proceedings of the 2016 TRECVID Ad-Hoc Video Search Task, Proceedings of the 2016 TRECVID Ad-Hoc Video Search TaskNIST, Gaithersburg, MD, USA.
VideoSketcher: Innovative Query Modes for Searching Videos through Sketches, Motion and Sound
Dupont Stéphane, Altiok Ozan Can, Bumin Aysegül, Dikmen Ceren, Giangreco Ivan, Heller Silvan, Külah Emre, Pironkov Gueorgui, Rossetto Luca, Sahillioglu Yusuf, Schuldt Heiko, Seddati Omar, Setinkaya Yusuf, Sezgin Metin, Tanase Claudiu, Toyan Emre, Wood Sean, Yeke Doguhan (2016), VideoSketcher: Innovative Query Modes for Searching Videos through Sketches, Motion and Sound, Proc. the 11th International one‐month Summer Workshop on Multimodal Interfaces (eNTERFACE’2015), Mons, BE.
vitrivr - A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections
Rossetto Luca, Giangreco Ivan, Tanase Claudiu, Schuldt Heiko (2016), vitrivr - A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections, in Proceedings of the 2016 ACM on Multimedia Conference, Amsterdam, NLACM, New York, NY, USA.
IMOTION - a Content-based Video Retrieval Engine
Rossetto Luca, Giangreco Ivan, Schuldt Heiko, Dupont Stéphane, Seddati Omar, Sezgin Metin, Sahillioğlu Yusuf (2015), IMOTION - a Content-based Video Retrieval Engine, in Proceedings of the 21st MultiMedia Modelling Conference (MMM2015) - Video Search Showcase Track, Sydney, AustraliaSpringer, Heidelgerg, Germany.
OSVC ‐ Open Short Video Collection 1.0
Rossetto Luca, Giangreco Ivan, Schuldt Heiko (2015), OSVC ‐ Open Short Video Collection 1.0, Technischer Bericht, Universität Basel, Basel.
ADAM — A Database and Information Retrieval System for Big Multimedia Collections
Giangreco Ivan, Al Kabary Ihab, Schuldt Heiko (2014), ADAM — A Database and Information Retrieval System for Big Multimedia Collections, in Proceedings of the 3rd International Congress on Big Data,, Anchorage, AK, USAIEEE, New York, NY, USA.
ADAM — A System for Jointly Providing IR and Database Queries in Large-Scale Multimedia Retrieval
Giangreco Ivan, Al Kabary Ihab, Schuldt Heiko (2014), ADAM — A System for Jointly Providing IR and Database Queries in Large-Scale Multimedia Retrieval, in Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information , Gold Coast, AustraliaACM, New York, NY, USA.
Cineast: A Multi-Feature Sketch-Based Video Retrieval Engine
Rossetto Luca, Giangreco Ivan, Schuldt Heiko (2014), Cineast: A Multi-Feature Sketch-Based Video Retrieval Engine, in Proceedings of the 16th IEEE International Symposium on Multimedia (ISM2014), Taichung, TaiwanIEEE, New York, NY, USA.
Crowd-based Semantic Event Detection and Video Annotation for Sports Videos
Sulser Fabio, Giangreco Ivan, Schuldt Heiko (2014), Crowd-based Semantic Event Detection and Video Annotation for Sports Videos, in Proceedings of the 3rd International ACM Workshop on Crowdsourcing for Multimedia, Orlando, FL, USAACM, New York, NY, USA.
Hey, vitrivr! - A Multimodal UI for Video Retrieval
Goel Prateek, Giangreco Ivan, Rossetto Luca, Tănase Claudiu, Schuldt Heiko, Hey, vitrivr! - A Multimodal UI for Video Retrieval, in Proceedings of the 39th European Conference on Information Retrieval (ECIR 2017), Aberdeen, Scotland, UKSpringer, Heidelberg, Germany.

Zusammenarbeit

Gruppe / Person Land
Formen der Zusammenarbeit
Distributed Little Red Hen Lab (https://sites.google.com/site/distributedlittleredhen/home) Vereinigte Staaten von Amerika (Nordamerika)
- vertiefter/weiterführender Austausch von Ansätzen, Methoden oder Resultaten
Bundesamt für Sport (BaSpo) Schweiz (Europa)
- vertiefter/weiterführender Austausch von Ansätzen, Methoden oder Resultaten
The Media Ecology Project, Dartmouth College Vereinigte Staaten von Amerika (Nordamerika)
- vertiefter/weiterführender Austausch von Ansätzen, Methoden oder Resultaten

Wissenschaftliche Veranstaltungen

Aktiver Beitrag

Titel Art des Beitrags Titel des Artikels oder Beitrages Datum Ort Beteiligte Personen
Google Summer of Code 2016 Mentor Summit Poster The vitrivr System 28.10.2016 Sunnyvale, CA, Vereinigte Staaten von Amerika Tanase Claudiu; Giangreco Ivan;
2016 ACM Multimedia Conference Vortrag im Rahmen einer Tagung vitrivr - A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections 15.10.2016 Amsterdam, Niederlande Tanase Claudiu; Rossetto Luca; Giangreco Ivan;
21st ACM International Conference on Intelligent User Interfaces (IUI'16) Vortrag im Rahmen einer Tagung Semantic Sketch-Based Video Retrieval with Autocompletion 07.03.2016 Sonoma, CA, Vereinigte Staaten von Amerika Tanase Claudiu;
22nd International Conference on Multimedia Modeling (MMM 2016) Poster Dealing with ambiguous Queries in Multimodal Video Retrieval 04.01.2016 Miami, FL, Vereinigte Staaten von Amerika Rossetto Luca; Tanase Claudiu; Giangreco Ivan;
22nd International Conference on Multimedia Modeling (MMM 2016) Vortrag im Rahmen einer Tagung IMOTION ‐ Searching for Video Sequences using Multi‐Shot Sketch Queries 04.01.2016 Miami, FL, Vereinigte Staaten von Amerika Rossetto Luca; Tanase Claudiu; Giangreco Ivan;
22nd International Conference on Multimedia Modeling (MMM 2016) Vortrag im Rahmen einer Tagung Searching in Video Collections using Sketches and Sample Images ‐ The Cineast System 04.01.2016 Miami, FL, Vereinigte Staaten von Amerika Rossetto Luca; Giangreco Ivan; Tanase Claudiu;
22nd International Conference on Multimedia Modeling (MMM 2016) Vortrag im Rahmen einer Tagung iAutoMotion ‐ an Autonomous Content‐based Video Retrieval Engine 04.01.2016 Miami, FL, Vereinigte Staaten von Amerika Rossetto Luca; Tanase Claudiu; Giangreco Ivan;
Microsoft Research, PhD Summer School Poster Cineast: Multi-modal image and video retrieval 26.06.2015 Cambridge, Grossbritannien und Nordirland Rossetto Luca;
Microsoft Research, PhD Summer School Poster ADAM: a database for multimedia retrieval 26.06.2015 Cambridge, Grossbritannien und Nordirland Giangreco Ivan;
21st International Conference on Multimedia Modeling (MMM 2015) Vortrag im Rahmen einer Tagung IMOTION ‐ a Content‐based Video Retrieval Engine 05.01.2015 Sydney, Australien Rossetto Luca; Schuldt Heiko; Giangreco Ivan;
16th IEEE International Symposium on Multimedia (ISM2014) Vortrag im Rahmen einer Tagung Cineast: A Multi‐Feature Sketch‐Based Video Retrieval Engine 10.12.2014 Taichung, Taiwan Rossetto Luca;
3rd International ACM Workshop on Crowdsourcing for Multimedia Vortrag im Rahmen einer Tagung Crowd‐based Semantic Event Detection and Video Annotation for Sports Videos 07.11.2014 Orlando, FL, Vereinigte Staaten von Amerika Giangreco Ivan;
37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR ’14) Vortrag im Rahmen einer Tagung ADAM — A System for Jointly Providing IR and Database Queries in Large‐Scale Multimedia Retrieval 06.07.2014 Gold Coast, Australien Giangreco Ivan; Schuldt Heiko;


Selber organisiert

Titel Datum Ort
eNTERFACE 2015 10.08.2015 Mons, Belgien

Kommunikation mit der Öffentlichkeit

Kommunikation Titel Medien Ort Jahr
Medienarbeit: Printmedien, Online-Medien "vitrivr": Bild- und Videosuche per Handskizze Pressetext International 2016
Medienarbeit: Printmedien, Online-Medien Afbeeldingen zoeken met een simpele schets De Ingenieur International 2016
Medienarbeit: Printmedien, Online-Medien Bild- und Videosuche per Handskizze Informatik.ch Deutschschweiz 2016
Referate/Veranstaltungen/Ausstellungen Die Stecknadel im Heuhaufen: Unter tausenden digitaler Fotos schnell das richtige finden (Open Day) Deutschschweiz 2016
Medienarbeit: Radio, Fernsehen Ein Bild sucht mehr als tausend Worte Deutschlandfunk International 2016
Medienarbeit: Printmedien, Online-Medien Forscher enwickeln Programm, das Bilder anhand von Skizzen findet Aargauer Zeitung Deutschschweiz 2016
Medienarbeit: Printmedien, Online-Medien In a new method for searching image databases, a hand-drawn sketch is all it takes phys.org International 2016
Medienarbeit: Printmedien, Online-Medien In a New Method for Searching Image Databases, a Hand-drawn Sketch Is all it Takes Innovations Report International 2016
Referate/Veranstaltungen/Ausstellungen Infotag Universität Basel, 2016 Deutschschweiz 2016
Medienarbeit: Printmedien, Online-Medien Lauter bekannte Gesichter Hannoversche Allgemeine Zeitung International 2016
Video/Film Natural Language Query Interface for vitrivr International 2016
Neue Medien (Web, Blogs, Podcasts, NewsFeed, usw.) Neues Suchen in Bilddatenbanken: Skizze von Hand genügt UniNews, Universität Basel Deutschschweiz 2016
Medienarbeit: Printmedien, Online-Medien Online-Suche mittels Skizzen 20min.ch Deutschschweiz 2016
Medienarbeit: Printmedien, Online-Medien So funktioniert die Bild- oder Videosuche mit vitrivr scinexx.de International 2016
Medienarbeit: Printmedien, Online-Medien Suchmaschine für Skizzen sem.seo.at International 2016
Medienarbeit: Printmedien, Online-Medien Technology: a new method of image search is a major draw! Asia Times International 2016
Video/Film vitrivr International 2016
Medienarbeit: Printmedien, Online-Medien Vitrivr is an open source engine that lets you search for videos with a sketch Daily News and Analyzes India International 2016
Referate/Veranstaltungen/Ausstellungen Die Stecknadel im Heuhaufen: Unter tausenden digitaler Fotos schnell das richtige finden (VHS Basel) Deutschschweiz 2015
Referate/Veranstaltungen/Ausstellungen Infotag Universität Basel, 2015 Deutschschweiz 2015
Neue Medien (Web, Blogs, Podcasts, NewsFeed, usw.) Big Data tamed with the Cloud Blog Microsoft Research International 2014
Medienarbeit: Printmedien, Online-Medien Videos via Suchanfrage durchsuchen Netzwoche Deutschschweiz 2014

Auszeichnungen

Titel Jahr
Winner of the 2017 Video Browser Showdown (VBS 2017), held at the Multimedia Modeling Conference (MMM 2017). 2017
Best Demo Award für: [RGH+ 16] Luca Rossetto, Ivan Giangreco, Silvan Heller, Claudiu Tănase, Heiko Schuldt. Searching in Video Collections using Sketches and Sample Images ‐ The Cineast System. In: Proceedings of the 22nd International Conference on Multimedia Modeling (MMM 2016), Miami, FL, USA, January 2016. Springer LNCS, Volume 9517, pp. 336‐341. http://link.springer.com/chapter/10.1007%2F978‐3‐ 319‐27674‐8_30. 2016

Anwendungsorientierte Outputs

Software

Name Jahr
vitrivr 2016


Verbundene Projekte

Nummer Titel Start Förderungsinstrument
137944 MM-DocTable: Multimedia Document Engineering Workflows on Tabletop Devices 01.10.2011 Projektförderung (Abt. I-III)

Abstract

Video is increasingly gaining importance as medium to capture and disseminate information. This is not only the case for personal use but also –and most importantly– for professional and educational applications. With the enormous growth of video collections, effective yet efficient content-based retrieval of (parts of) videos is becoming more and more essential. Conventionally, video retrieval relies on metadata such as manual annotations, or inherent features extracted from the video. However, the most decisive information that distinguishes video content from static content, the movement of individual objects across subsequent frames, so far is largely ignored. This is particularly the case for so-called augmented video where additional spatio-temporal data on the movement of objects (e.g., captured by dedicated sensor systems) is available in addition to the actual video content. The IMOTION project will develop and evaluate innovative multi-modal user interfaces for interacting with augmented videos. Starting with an extension of existing query paradigms (keyword search in manual annotations), image search (query by example in key frames), IMOTION will consider novel sketch- and speech-based user interfaces. In particular, novel types of motion queries will be supported where users can specify motion paths of objects, via sketches, gestures, natural language interfaces, or combinations thereof. Several types of user interfaces (voice, tablets, multi-touch tables, interactive paper) will be supported and seamlessly combined so as to smoothly migrate a session from one type of user interface to another during the process of specifying and refining a query. This will be based on novel approaches to representation learning and the extraction of high-level motion descriptors from augmented videos, based on a motion ontology. In addition, IMOTION will develop novel index structures that jointly support traditional video features and the additional motion metadata. A major contribution will be the quantitative and qualitative evaluation and user studies of the intelligent multi-modal interfaces and query paradigms developed in two concrete use cases – sample applications from which the project will select include, but are not limited to, augmented sports videos where users search on the basis of trajectories of player or ball movements, educational videos from the natural sciences where users search for animal movements inside a horde or a swarm, or sketch-based searches for currents in the sea captured by sensors integrated into buys. The IMOTION consortium will openly publish the augmented video collections and the motion metadata created in the course of the project’s evaluation activities.
-