Projekt

Zurück zur Übersicht

MM-DocTable: Multimedia Document Engineering Workflows on Tabletop Devices

Titel Englisch MM-DocTable: Multimedia Document Engineering Workflows on Tabletop Devices
Gesuchsteller/in Schuldt Heiko
Nummer 137944
Förderungsinstrument Projektförderung (Abt. I-III)
Forschungseinrichtung Fachbereich Informatik Departement Mathematik und Informatik Universität Basel
Hochschule Universität Basel – BS
Hauptdisziplin Informatik
Beginn/Ende 01.10.2011 - 30.09.2014
Bewilligter Betrag 286'873.00
Alle Daten anzeigen

Keywords (6)

Similarity Search; Pen and Touch Interfaces; Human-Computer Interaction; Content-based Multimedia Retrieval; Multimedia Document Production Workflows; Digital Tabletops

Lay Summary (Englisch)

Lead
Lay summary

Multi-touch devices have recently enjoyed a surge of popularity by supported by an ever-increasing wealth of applications which is a testimony that digital natural user interfaces (NUI) have gained attraction in the public. While paper is still far from extinct, we observe that habits and interaction patterns are more and more shifting towards those types of digital devices, as they become user-friendlier and more effective at performing regular document tasks. Digital interactive tabletops form an important subcategory, which receives considerable attention in the different HCI-research communities, yet surprisingly has not seen a great number of advanced document engineering applications that take advantage of this type of platform. We postulate that bimanual pen-and-touch-operated tabletops, as enhanced virtual office desks, are very suitable platforms to perform a number of document-engineering tasks and are a natural extension of the paper-digital interface paradigm, the obvious advantage of tabletops being that they provide immediate, interactive feedback to users upon their actions. Currently, though, systems only demonstrate the novel UI concepts in rather isolated, single-document environments and are not yet integrated in any particular document engineering workflow. Very often, documents are engaged in a process involving several interactions with interfaces of sometimes heterogeneous systems and so the need to integrate tabletop UIs with a variety of legacy workflows arises.

The advent of NUIs has also strongly influenced the multimedia retrieval community. In particular, novel approaches that make use of the interaction capabilities of NUIs for posing queries, such as query by sketch applied to digital image collections, have become increasingly popular. While the general idea to search for images on the basis of a (rough) user-provided sketch is very appealing, it is currently limited to a particular search task, namely known-item search. With the availability of large digital video collections, query by sketch will also become highly relevant for content-based video retrieval. However, in order to go beyond the comparison of a sketch with a single video frame, support for dedicated gestures to express the motion of objects over several frames are needed.

The MM-DocTable project aims at providing a sound architecture and tools to support document engineering tasks performed on digital tabletops in a wider range of contexts, in particular by enabling rapid and seamless access to reference material for multimedia document creation or editing.  Most importantly, the proposed architecture will support several querying methods (keyword search, query by example, and query by sketch) with a view to apply them interchangeably in a number of retrieval scenarios. The pen-and-touch-operated UI of the tabletop will be designed so that the different retrieval operations are all seamlessly executable within the main document task, that is, without requiring tedious context switches.

To achieve these goals, the project will consist of four main parts. First, we will investigate what scenarios can benefit from complex document engineering workflows on tabletop devices that jointly use different query paradigms. Second, we will develop new algorithms for the detection of prominent objects in sketch-based image retrieval in order to apply query by sketch to other interaction intentions beyond known item search. Third, we will extend sketch-based retrieval to digital video collections, by adding gestures for specifying the motion of objects. Fourth, we will seamlessly integrate all these services into the tabletop UI so that users are able to directly access elements of the multimedia database and use this integrated system in user studies.

Direktlink auf Lay Summary Letzte Aktualisierung: 21.02.2013

Verantw. Gesuchsteller/in und weitere Gesuchstellende

Mitarbeitende

Publikationen

Publikation
ADAM — A Database and Information Retrieval System for Big Multimedia Collections
Giangreco Ivan, Al Kabary Ihab, Schuldt Heiko (2014), ADAM — A Database and Information Retrieval System for Big Multimedia Collections, in Proceedings of the 3rd International Congress on Big Data, Anchorage, AK, USAIEEE, New York, NY, USA.
Enhancing Sketch-based Sport Video Retrieval by Suggesting Relevant Motion Paths
Al Kabary Ihab, Schuldt Heiko (2014), Enhancing Sketch-based Sport Video Retrieval by Suggesting Relevant Motion Paths, in Proceedings of the 37th ACM SIGIR Conference, Gold Coast, AustraliaACM, New York, NY, USA.
Spatial Querying of Geographical Data with Pen-Input Scopes
Matulic Fabrice, Caspar David, Norrie Moira (2014), Spatial Querying of Geographical Data with Pen-Input Scopes, in Proc. ACM Interactive Tabletops and Surfaces 2014 Conference (ITS 2014), Dresden, GermanyACM, New York, NY, USA.
Spatial Querying of Geographical Data with Pen-Input Scopes
Matulic Fabrice, Caspar David, Norrie Moira (2014), Spatial Querying of Geographical Data with Pen-Input Scopes, in Proceedings of the ACM Interactive Tabletops and Surfaces Conference (ITS 2014), Dresden, GermanyACM, New York, NY, USA.
Using Hand Gestures for Specifying Motion Path Queries in Sketch-based Video Retrieval
Al Kabary Ihab, Schuldt Heiko (2014), Using Hand Gestures for Specifying Motion Path Queries in Sketch-based Video Retrieval, in Proceedings of the 36th European Conference on Information Retrieval (ECIR 2014), Amsterdam, The NetherlandsSpringer, Heidelberg, Germany.
Beyond WIMP: Designing NUIs to Support Productivity Document Tasks
Matulic Fabrice (2013), Beyond WIMP: Designing NUIs to Support Productivity Document Tasks, in Blended Interaction, Envisioning Future Collaborative Interactive Spaces, CHI 2013 Workshop, Paris, FranceACm, New York, NY, USA.
Gesture-Supported Document Creation on Pen and Touch Tabletops
Matulic Fabrice, Norrie Moira C., Al Kabary Ihab, Schuldt Heiko (2013), Gesture-Supported Document Creation on Pen and Touch Tabletops, in Proceedings of the 31st ACM Conference on Human Factors in Computing Systems (CHI 2013), Prais, FranceACM, New York, NY, USA.
Pen and touch gestural environment for document editing on interactive tabletops
Matulic Fabrice, Norrie Moira C. (2013), Pen and touch gestural environment for document editing on interactive tabletops, in Proceedings of the ACM Interactive Tabletops and Surfaces Conference (ITS 2013), St. Andrews, UKACM, New York, NY, USA.
QUEST: Towards a Multi-Modal CBIR Framework Combining Query-by-Example, Query-by-Sketch, and Text Search
Al Kabary Ihab, Giangreco Ivan, Schuldt Heiko, Matulic Fabrice, Norrie Moira (2013), QUEST: Towards a Multi-Modal CBIR Framework Combining Query-by-Example, Query-by-Sketch, and Text Search, in Proc. of the 9th IEEE Int'l Workshop on Multimedia Information Processing and Retrieval (MIPR 2013), Anaheim, CA, USAIEEE, New York, NY, USA.
SportSense: Using Motion Queries to Find Scenes in Sports Videos
Al Kabary Ihab, Schuldt Heiko (2013), SportSense: Using Motion Queries to Find Scenes in Sports Videos, in Proc. of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013), San Francisco, CA, USAACM, New York, NY, USA.
TOUCHify: Bringing Pen-Based Touch Screen Functionality to Flat Panel Display Screens
Al Kabary Ihab, Büchler Marcel, Schuldt Heiko (2013), TOUCHify: Bringing Pen-Based Touch Screen Functionality to Flat Panel Display Screens, in Proceedings of the International Conference on Information Society (i-Society 2013), Toronto, CanadaIEEE, New York, NY, USA.
Towards Sketch-based Motion Queries in Sports Videos
Al Kabary Ihab, Schuldt Heiko (2013), Towards Sketch-based Motion Queries in Sports Videos, in Proceedings of the 15th IEEE International Symposium on Multimedia (ISM 2013), Anaheim, CA, USAIEEE, New York, NY, USA.
A User Interface for Query-by-Sketch based Image Retrieval with Color Sketches
Giangreco Ivan, Springmann Michael, Al Kabary Ihab, Schuldt Heiko (2012), A User Interface for Query-by-Sketch based Image Retrieval with Color Sketches, in Proceedings of the 34th European Conference on Information Retrieval, Barcelona, SpainSpringer, Heidelberg, Germany.
An Interactive Paper and Digital Pen Interface for Query-by-Sketch Image Retrieval
Kreuzer Roman, Springmann Michael, Al Kabary Ihab, Schuldt Heiko (2012), An Interactive Paper and Digital Pen Interface for Query-by-Sketch Image Retrieval, in Proceedings of the 34th European Conference on Information Retrieval (ECIR’12), Barcelona, SpainSpringer, Heidelberg, Germany.
Sketch-based Image Similarity Search with a Pen and Paper Interface
Al Kabary Ihab, Schuldt Heiko (2012), Sketch-based Image Similarity Search with a Pen and Paper Interface, in Proceedings of the 35th International Conference on Research and Development in Information Retrieva, Portland, OR, USAACM, New York, NY, USA.
SKETCHify - an Adaptive Prominent Edge Detection Algorithm for Optimized Query-by-Sketch Image Retrieval
Al Kabary Ihab, Schuldt Heiko (2012), SKETCHify - an Adaptive Prominent Edge Detection Algorithm for Optimized Query-by-Sketch Image Retrieval, in Proceedings of the 10th International Workshop on Adaptive Multimedia Retrieval (AMR'12), Copenhagen, DenmarkSpringer, Heidelberg, Germany.
Supporting Active Reading on Pen and Touch-Operated Tabletops
Matulic Fabrice, Norrie Moira (2012), Supporting Active Reading on Pen and Touch-Operated Tabletops, in Procc. Intl. Working Conference on Advanced Visual Interfaces (AVI 2012), Capri Island, ItalyACM, New York, NY, USA.
Empirical Evaluation of Uni- and Bimodal Pen and Touch Interaction
Matulic Fabrice, Norrie Moira, Empirical Evaluation of Uni- and Bimodal Pen and Touch Interaction, in Proc. ACM Interactive Tabletops and Surfaces Conference (ITS 2012), Cambridge (MA), USAACM, New York, NY, USA.

Zusammenarbeit

Gruppe / Person Land
Formen der Zusammenarbeit
Free University of Brussels Belgien (Europa)
- vertiefter/weiterführender Austausch von Ansätzen, Methoden oder Resultaten
Koç University Türkei (Europa)
- vertiefter/weiterführender Austausch von Ansätzen, Methoden oder Resultaten
University of Mons Belgien (Europa)
- vertiefter/weiterführender Austausch von Ansätzen, Methoden oder Resultaten

Wissenschaftliche Veranstaltungen

Aktiver Beitrag

Titel Art des Beitrags Titel des Artikels oder Beitrages Datum Ort Beteiligte Personen
37th ACM SIGIR Conference Poster Enhancing Sketch-based Sport Video Retrieval by Suggesting Relevant Motion Paths 06.07.2014 Gold Coast, Australien Schuldt Heiko;
3rd International Congress on Big Data Vortrag im Rahmen einer Tagung A Database and Information Retrieval System for Big Multimedia Collections 27.06.2014 Anchorage, AK, Vereinigte Staaten von Amerika Ihab Al Kabary;
36th European Conference on Information Retrieval (ECIR 2014) Poster Using Hand Gestures for Specifying Motion Queries in Sketch-based Video Retrieval 14.04.2014 Amsterdam, Niederlande Ihab Al Kabary;
9th IEEE International Workshop on Multimedia Information Processing and Retrieval (MIPR 2013) Vortrag im Rahmen einer Tagung QUEST: Towards a Multi-Modal CBIR Framework Combining Query-by-Example, Query-by-Sketch, and Text Search 09.12.2013 Anaheim, CA, Vereinigte Staaten von Amerika Ihab Al Kabary;
15th IEEE International Symposium on Multimedia (ISM 2013) Vortrag im Rahmen einer Tagung Towards Sketch-based Motion Queries in Sports Videos 09.12.2013 Anaheim, CA, Vereinigte Staaten von Amerika Ihab Al Kabary;
22nd ACM International Conference on Information and Knowledge Management (CIKM 2013) Vortrag im Rahmen einer Tagung SportSense: Using Motion Queries to Find Scenes in Sports Videos 27.10.2013 San Francisco, CA, USA, Vereinigte Staaten von Amerika Schuldt Heiko;
International Conference on Information Society (i-Society 2013) Vortrag im Rahmen einer Tagung TOUCHify: Bringing Pen-Based Touch Screen Functionality to Flat Panel Display Screens 24.06.2013 Toronto, Canada, Kanada Ihab Al Kabary;
31st ACM Conference on Human Factors in Computing Systems (CHI 2013) Poster Gesture-Supported Document Creation on Pen and Touch Tabletops 27.04.2013 Paris, France, Frankreich Norrie Moira;
10th International Workshop on Adaptive Multimedia Retrieval (AMR’12) Vortrag im Rahmen einer Tagung SKETCHify - an Adaptive Prominent Edge Detection Algorithm for Optimized Query-by-Sketch Image Retrieval 24.10.2012 Copenhagen, Denmark, Dänemark Ihab Al Kabary;
35th International Conference on Research and Development in Information Retrieval (SIGIR'12) Vortrag im Rahmen einer Tagung Sketch-based Image Similarity Search with a Pen and Paper Interface 12.08.2012 Portland, OR, USA, Vereinigte Staaten von Amerika Schuldt Heiko;
Proceedings of the 34th European Conference on Information Retrieval (ECIR 2012) Poster A User Interface for Query-by-Sketch based Image Retrieval with Color Sketches 01.04.2012 Barcelona, Spanien Ihab Al Kabary;
34th European Conference on Information Retrieval (ECIR’12) Vortrag im Rahmen einer Tagung An Interactive Paper and Digital Pen Interface for Query-by-Sketch Image Retrieval 01.04.2012 Barcelona, Spain, Spanien Ihab Al Kabary;


Kommunikation mit der Öffentlichkeit

Kommunikation Titel Medien Ort Jahr
Weitere Aktivitäten Tag der offenen Tür, Fachbereich Informatik Deutschschweiz 2012

Verbundene Projekte

Nummer Titel Start Förderungsinstrument
151571 IMOTION - Intelligent Multi-Modal Augmented Video Motion Retrieval System 01.01.2014 Resource not found: 'ecd1295e-ae98-42a1-b855-8c8f7a9c899c'
126829 PAD-IR: Paper-Digital System for Information Capture and Retrieval 01.10.2009 Projektförderung (Abt. I-III)

Abstract

The project is a continuation of SNF Project 200020_126829 PAD-IR: Paper-Digital System for Information Capture and Retrieval in which we investigated an information retrieval system that is able to truly bridge the paper-digital divide by allowing retrieval across different forms of media, including handwritten notes and sketches. In the PAD-IR project, we explored ways to retrieve documents based on queries specified digitally or on paper, or even some combination of both.In this continuation, we acknowledge the recent surge of popularity enjoyed by (multi-)touch devices such as tablets supported by an ever-increasing wealth of applications which is a testimony that digital natural user interfaces (NUI) have gained attraction in the public. While paper is still far from extinct, we observe that habits and interaction patterns are more and more shifting towards those types of digital devices, as they become user-friendlier and more effective at performing regular document tasks. In this family of NUI systems, digital interactive tabletops form an important subcategory, which receives considerable attention in the different HCI-research communities, yet surprisingly has not seen a great number of advanced document engineering applications that take advantage of this type of platform. We postulate that bimanual pen-and-touch-operated tabletops, as enhanced virtual office desks, are very suitable platforms to perform a number of document-engineering tasks and are a natural extension of the paper-digital interface paradigm, the obvious advantage of tabletops being that they provide immediate, interactive feedback to users upon their actions. Currently, though, systems only demonstrate the novel UI concepts in rather isolated, single-document environments and are not yet integrated in any particular document engineering workflow. Very often, documents are engaged in a process involving several interactions with interfaces of sometimes heterogeneous systems and so the need to integrate tabletop UIs with a variety of legacy workflows arises.The advent of NUIs has also strongly influenced the multimedia retrieval community. In particular, novel approaches that make use of the interaction capabilities of NUIs for posing queries, such as query by sketch applied to digital image collections, have become increasingly popular. While the general idea to search for images on the basis of a (rough) user-provided sketch is very appealing, it is currently limited to a particular search task, namely known-item search. With the availability of large digital video collections, query by sketch will also become highly relevant for content-based video retrieval. However, in order to go beyond the comparison of a sketch with a single video frame, support for dedicated gestures to express the motion of objects over several frames are needed.The MM-DocTables project aims at providing a sound architecture and tools to support document engineering tasks performed on digital tabletops in a wider range of contexts, in particular by enabling rapid and seamless access to reference material for multimedia document creation or editing. Because pen-and-touch tabletops and paper share similar interaction patterns, the sketch-based retrieval techniques developed for the latter during the course of the PAD-IR project can be easily applied to the former and even enhanced thanks to the richer interaction and feedback capabilities of the digital surface. Most importantly, the proposed architecture will support several querying methods (keyword search, query by example, and query by sketch) with a view to apply them interchangeably in a number of retrieval scenarios. The pen-and-touch-operated UI of the tabletop will be designed so that the different retrieval operations are all seamlessly executable within the main document task, that is, without requiring tedious context switches. To achieve these goals, the project will consist of four main parts. First, we will investigate what scenarios can benefit from complex document engineering workflows on tabletop devices that jointly use different query paradigms. Second, we will develop new algorithms for the detection of prominent objects in sketch-based image retrieval in order to apply query by sketch to other interaction intentions beyond known item search. Third, we will extend sketch-based retrieval to digital video collections, by adding gestures for specifying the motion of objects. Fourth, we will seamlessly integrate all these services into the tabletop UI so that users are able to directly access elements of the multimedia database and use this integrated system in user studies.
-