Project

Back to overview

SMILE: Scalable Multimodal sign language Technology for sIgn language Learning and assessmEnt

English title SMILE: Scalable Multimodal sign language Technology for sIgn language Learning and assessmEnt
Applicant Magimai-Doss Mathew
Number 160811
Funding scheme Sinergia
Research institution IDIAP Institut de Recherche
Institution of higher education Idiap Research Institute - IDIAP
Main discipline Information Technology
Start/End 01.03.2016 - 29.02.2020
Approved amount 1'200'000.00
Show all

All Disciplines (2)

Discipline
Information Technology
Educational science and Pedagogy

Keywords (5)

Sign language technology; Computer-/mobile assisted language testing; Sign language recognition; Common European Framework of Reference; Sign assessment

Lay Summary (German)

Lead
Ziel des Projektes SMILE ist die Entwicklung des ersten Gebärdensprachtestsystems für die Deutschschweizerische Gebärdensprache (DSGS), das sich automatischer Gebärdenspracherkennung bedient. Um dieses Ziel zu erreichen, verfolgt das Projekt einen multidisziplinären Ansatz, der zwei Forschungsstränge zusammenbringt, Gebärdensprachtechnologie und Gebärdensprachtesten, mit einer gemeinsamen Verbindung zur Gebärdensprachlinguistik.
Lay summary

Ziel des Projektes SMILE ist die Entwicklung des ersten Gebärdensprachtestsystems für die Deutschschweizerische Gebärdensprache (DSGS), das sich automatischer Gebärdenspracherkennung bedient. Um dieses Ziel zu erreichen, verfolgt das Projekt einen multidisziplinären Ansatz, der zwei Forschungsstränge zusammenbringt, Gebärdensprachtechnologie und Gebärdensprachtesten, mit einer gemeinsamen Verbindung zur Gebärdensprachlinguistik. Das Projektkonsortium setzt sich aus drei Partnerinstitutionen mit sich ergänzender Expertise zusammen:

  1. Das Idiap Research Institute (Martigny, Schweiz) wird das Projekt koordinieren und inhaltlich durch die Entwicklung eines neuartigen automatischen Gebärdensprachtest- und -feedbackansatzes, der durch einen Spracherkennungsansatz inspiriert ist, zum Projekt beitragen, .
  2. Die Interkantonale Hochschule für Heilpädagogik Zürich (HfH) wird ihre Expertise im Bereich Gebärdensprachtesten und Gebärdensprachlinguistik (in Zusammenarbeit mit dem Forschungszentrum für Gebärdensprache Basel, FZG) ins Projekt einbringen.
  3. Die Universität Surrey (Grossbritannien) wird ihre langjährige Expertise im Bereich Gebärdensprachtechnologie, visuelle Datenakquisition und Computer Vision und insbesondere ihre breite Erfahrung in aktueller Forschung zu Gebärdensprachtechnologie ins Projekt einbringen.

Das SMILE-Projekt wird Finanzierung für drei Doktoranden, zwei Postdoktoranden, drei wissenschaftliche Mitarbeiter und gehörlose Experten bereitstellen. Am Projekt werden auch gehörlose DSGS-Benutzer beteiligt sein. Das SMILE-Projekt folgt dem Ansatz des Gemeinsamen Europäischen Referenzrahmens für Sprachen (GER), indem es ein Testsystem entwickelt, das die Produktion von DSGS-Vokabular auf der Stufe A1 misst, unter erstmaligem Einbezug von neuen Technologien für Gebärdensprache. Das Projekt wird so eine Plattform für Lehr- und Lernsysteme bereitstellen, die spezifisch für DSGS ist, aber auch als Modell für andere Gebärdensprachen dienen kann.

Direct link to Lay Summary Last update: 19.02.2016

Responsible applicant and co-applicants

Employees

Publications

Publication
Data-Driven Movement Subunit Extraction from Skeleton Information for Modeling Signs and Gestures
Tornay Sandrine, Razavi Marzieh, Magimai-Doss Mathew (2019), Data-Driven Movement Subunit Extraction from Skeleton Information for Modeling Signs and Gestures, Idiap Research Institute, Idiap Internal Research Report Idiapl-RR-02-2019, Martigny.
Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs
Koller Oscar, Zargaran Sepehr, Ney Hermann, Bowden Richard (2018), Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs, in International Journal of Computer Vision, 126(12), 1311-1325.
Eine Untersuchung der Verwendungsfrequenz von Gebärdenvarianten bei L1- und L2-Benutzern im Rahmen des SMILE-Projektes
Arter Lisa (2018), Eine Untersuchung der Verwendungsfrequenz von Gebärdenvarianten bei L1- und L2-Benutzern im Rahmen des SMILE-Projektes, HfH, BA Thesis, Zurich.
Gebärdensprachtests in der Deutschschweiz
HaugTobias (2018), Gebärdensprachtests in der Deutschschweiz, in Hörgeschädigtenpädagogik, 72(4), 199-204.
Phonologische Produktionsfehler bei L2-Lernern der Deutschschweizer Gebärdensprache: Eine Analyse und Kategorisierung von phonologischen Produktionsfehlern im Rahmen des SMILE-Projektes
Rittiner Laura (2018), Phonologische Produktionsfehler bei L2-Lernern der Deutschschweizer Gebärdensprache: Eine Analyse und Kategorisierung von phonologischen Produktionsfehlern im Rahmen des SMILE-Projektes, HfH - BA Thesis, Zurich.
Unterschiede von Produktionsfehlern in Bezug auf die manuellen Komponenten bei L2-Benutzern des SMILE-Projekts
Schlumpf Cheryl (2018), Unterschiede von Produktionsfehlern in Bezug auf die manuellen Komponenten bei L2-Benutzern des SMILE-Projekts, HfH, BA Thesis, Zurich.
Particle Filter Based Probabilistic Forced Alignment for Continuous Gesture Recognition
Camgoz Necati Cihan, Hadfield Simon, Bowden Richard (2017), Particle Filter Based Probabilistic Forced Alignment for Continuous Gesture Recognition, in Proc. Chalearn 2017, IEEE International Conference on Computer Vision Workshops (ICCVW), IEEE, Venice, Italy.
SubUNets: End-to-end Hand Shape and Continuous Sign Language Recognition
Camgoz Necati Cihan, Hadfield Simon, Koller Oscar, Bowden Richard (2017), SubUNets: End-to-end Hand Shape and Continuous Sign Language Recognition, in Proceedings of IEEE Int. Conf. Computer Vision (ICCV), IEEE, Venice.
TMAGIC: A Model-free 3D Tracker
Lebeda Karel, Hadfield Simon, Bowden Richard (2017), TMAGIC: A Model-free 3D Tracker, in IEEE Transactions on Image Processing, 26(9), 4378-4388.
Stereo Reconstruction Using Top-down Cues. Computer Vision and Image Understanding
Hadfield S, Lebeda K, Bowden R (2017), Stereo Reconstruction Using Top-down Cues. Computer Vision and Image Understanding, in Computer Vision and Image Understanding, 157, 206-222.
Development and Evaluation of Two Vocabulary Tests for Swiss German Sign Language
Haug Tobias (2017), Development and Evaluation of Two Vocabulary Tests for Swiss German Sign Language, University of Lancaster, Masters Thesis, UK.
Hollywood 3D: What are the best 3D features for Action Recognition?
Hadfield S, Lebeda K, Bowden R (2017), Hollywood 3D: What are the best 3D features for Action Recognition?, in International Journal of Computer Vision, 121(1), 95-110.
Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition
Koller O, Zargaran S, Ney H, Bowden R (2016), Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition, in Proceedings of British Machine Vision Conference, York, UKBritish Machine Vision Association (BMVA), BMVA Press.
Direct-from-Video: Unsupervised NRSfM
Lebeda K, Hadfield S, Bowden R (2016), Direct-from-Video: Unsupervised NRSfM, in Proceedings of the ECCV workshop on Recovering 6D Object Pose Estimation, ECCV, ECCV.
Selecting items for the DSGS vocabulary production test
Ebling S, Boyes Braem P, Tissi K, Sidler-Miserez S, Haug T (2016), Selecting items for the DSGS vocabulary production test, Interkantonale Hochschule für Heilpädagogik, HfH, Zurich.
Using Convolutional 3D Neural Networks for User-Independent Continuous Gesture Recognition
Camgoz N C, Hadfield S, Koller O, Bowden R (2016), Using Convolutional 3D Neural Networks for User-Independent Continuous Gesture Recognition, in Proceedings of IEEE International Conference of Pattern Recognition (ICPR), ChaLearn Workshop, 2016, Cancun, MexicoIAPR, IEEE, IEEEXplore.
Automatic sign language recognition for sign language assessment
HaugTobias, EblingSarah, Boyes BraemPenny, TissiKatja, Sidler-MiserezSandra, Automatic sign language recognition for sign language assessment, in Conference on Technology-Based Language Assessment 2018, European Association for Language Testing and Assessment (EALTA), Bochum, Germany.
HARD-PnP: PnP Optimization Using a Hybrid Approximate Representation
Hadfield Simon James, Lebeda Karel, Bowden Richard, HARD-PnP: PnP Optimization Using a Hybrid Approximate Representation, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(3), 768-774.
HMM Based Approaches to Model Multichannel Information in Sign Language Inspired from Articulatory Features-based Speech Processing
Tornay Sandrine, Razavi M, Camgoz Necati Cihan, Bowden Richard, Magimai-Doss Mathew, HMM Based Approaches to Model Multichannel Information in Sign Language Inspired from Articulatory Features-based Speech Processing, in In Proc. ICASSP 2019, In Proc. ICASSP 2019, UK.
Neural Sign Language Translation
Camgoz Necati Cihan, Hadfield Simon, Koller Oscar, Ney Hermann, Bowden Richard, Neural Sign Language Translation, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR18), IEEE, Salt Lake City.
Sign Language Learning and Assessment in German Switzerland: Exploring the potential of vocabulary size tests for Swiss German Sign Language.
Haug Tobias, Ebling Sarah, Boyes Braem Penny, Sidler-Miserez Sandra, Tissi Katja, Sign Language Learning and Assessment in German Switzerland: Exploring the potential of vocabulary size tests for Swiss German Sign Language., in Language Education & Assessment.
Sign Language Production using Neural Machine Translation and Generative Adversarial Networks
Stoll Stephanie, Camgoz Necati Cihan, Hadfield Simon, Bowden Richard, Sign Language Production using Neural Machine Translation and Generative Adversarial Networks, in British Machine Vision Conference, British Machine Vision Conference, Newcastle, UK.
SMILE Swiss German Sign Language Dataset
Ebling Sarah, Camgöz Necati Cihan, Boyes Braem Penny, Tissi Katja, Sidler-Miserez Sandra, Stoll Stephanie, Hadfield Simon, Haug Tobias, Bowden Richard, Tornay Sandrine, Razavi Marzieh, Magimai-Doss Mathew, SMILE Swiss German Sign Language Dataset, in Proceedings of 11th International Conference on Language Resources and Evaluation (LREC 2018), ELRA, Japan.
Use of new technologies in L2 sign language assessment
Ebling Sarah, Camgoz Necati Cihan, Bowden Richard, Use of new technologies in L2 sign language assessment, Oxford: OUP, Oxford.
Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos
Koller Oscar, Camgoz Necati Cihan, Hermann Ney, Bowden Richard, Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos, in Transactions on Pattern Analysis and Machine Intelligence, IEEE, UK.
What’s wrong? Rethinking the concept of 'citation forms'
Tissi Katja, Sidler-Miserez Sandra, Ebling Sarah, Boyes Braem Penny, What’s wrong? Rethinking the concept of 'citation forms', in International Conference on Sign Language Acquisition (ICSLA), International Conference on Sign Language Acquisition (ICSLA), Istanbul, Turkey.

Collaboration

Group / person Country
Types of collaboration
DCAL Deafness Cognition and Language Research Centre – UCL Great Britain and Northern Ireland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
University of Oxford Great Britain and Northern Ireland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
RWTH AACHEN UNIVERSITY Germany (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
DCAL University College London Great Britain and Northern Ireland (Europe)
- in-depth/constructive exchanges on approaches, methods or results

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
Demo to industry Poster demo 05.04.2019 Surrey, Great Britain and Northern Ireland Bowden Richard;
Forschung und Entwicklung in der Fremdsprachendidaktik Talk given at a conference Modalitätsspezifische Aspekte des Gebärdensprachlernens und -testens 25.02.2019 Fribourg, Switzerland Tissi Katja; Haug Tobias; Ebling Sarah;
EDEE PhD students day Poster Phonology-based sign language recognition and assessment 30.11.2018 Lausanne, Switzerland Tornay Sandrine;
Second Swiss Conference on Barrier-Free Communication (BFC) 2018 Talk given at a conference Sign language technology in educational settings 10.11.2018 University of Geneva, Switzerland Ebling Sarah;
EPSRC Camera Expert Speaker series Talk given at a conference Keynote 25.09.2018 Bath, Great Britain and Northern Ireland Bowden Richard;
Nationale Tagung Netzwerk Forschung Sonderpädagogik Talk given at a conference Sprachtechnologie als Beitrag zur Barrierefreiheit 04.09.2018 University of Applied Sciences of Special Needs Education (HfH), Zurich, Switzerland Ebling Sarah;
Forschungskolloquium, PH St. Gallen Talk given at a conference Methodische Herausforderungen bei der Entwicklung und Evaluation von Tests zur Deutschschweizerischen Gebärdensprache 19.08.2018 St. Gallen, Switzerland Haug Tobias;
Controlo2018 conference Talk given at a conference Keynote 06.06.2018 Ponta Delgada, Portugal Bowden Richard;
Universität Innsbruck, Institut für Fachdidaktik Talk given at a conference Automatische Gebärdenspracherkennung und Gebärdensprachtests 24.04.2018 Innsbruck, Austria Haug Tobias;
ECML project network meeting "ProSign 2: Promoting excellence in sign language instruction" Talk given at a conference Rater training and scoring issues 05.04.2018 Graz, Austria Haug Tobias; Tissi Katja;
Indian Institute of Science Talk given at a conference Towards Phonologically Motivated Sign Language Processing Talk 03.04.2018 Banglore, India Magimai-Doss Mathew;
International Conference on Computer Vision Talk given at a conference SubUNets: End-to-end Hand Shape and Continuous Sign Language Recognition 22.10.2017 Venice, Italy Bowden Richard; Camgoz Necati Cihan;
2016 ChaLearn Looking at People Workshop ICPR Talk given at a conference Using Convolutional 3D Neural Networks for User-Independent Continuous Gesture Recognition 04.12.2016 Cancun, Mexico Bowden Richard;


Knowledge transfer events

Active participation

Title Type of contribution Date Place Persons involved
Demo for industry Talk 05.04.2019 Surrey, Great Britain and Northern Ireland Bowden Richard; Camgoz Necati Cihan; Mendez Maldonaldo Oscar; Hadfield Simon;
CVSSP 30th Anniversary evening showcase Performances, exhibitions (e.g. for education institutions) 04.04.2019 Surrey, Great Britain and Northern Ireland Camgoz Necati Cihan; Bowden Richard; Mendez Maldonaldo Oscar; Hadfield Simon;
University of Surrey Annual Review Performances, exhibitions (e.g. for education institutions) 30.01.2019 Surrey, Great Britain and Northern Ireland Mendez Maldonaldo Oscar;
ICT for Inclusion Talk 26.09.2018 St. Gallen, Switzerland Ebling Sarah;
Informatiktage Talk 17.06.2017 Zurich, Switzerland Ebling Sarah;
Menschen mit Behinderung in der Welt von morgen Poster 16.06.2017 Zurich, Switzerland Ebling Sarah;
ICT and Special Education Talk 11.05.2017 Bern, Switzerland Ebling Sarah;


Self-organised

Title Date Place
SMILE information event 28.02.2018 Zurich, Switzerland

Communication with the public

Communication Title Media Place Year
Media relations: print media, online media How AI could help you learn sign language The conversation International 2019
Media relations: print media, online media Quand la voix ouvre en grand les portes de la technologie Le Matin Dimanche Western Switzerland 2019
Media relations: print media, online media Researchers are creating an interactive ‘computer game’ that helps you learn sign language Independant International 2019
Talks/events/exhibitions Automatische Gebärdenspracherkennung und Gebärdensprachtests International 2018
Talks/events/exhibitions HfH Event, Demo to the deaf community German-speaking Switzerland 2018
Talks/events/exhibitions Journée Futur en tous genres Western Switzerland 2018
Talks/events/exhibitions Methodische Herausforderungen bei der Entwicklung und Evaluation von Tests zur Deutschschweizerische German-speaking Switzerland 2018
Talks/events/exhibitions SMILE Showcase event German-speaking Switzerland 2018
New media (web, blogs, podcasts, news feeds etc.) Projekt SMILE Youtube International Western Switzerland Rhaeto-Romanic Switzerland Italian-speaking Switzerland German-speaking Switzerland 2016

Awards

Title Year
Fellowship of the International Association of Pattern Recognition (FIAPR) in Dec 2017 in recognition of “ For contributions to computer vision in the fields of sign language, gesture and activity recognition and service to IAPR “ 2017

Associated projects

Number Title Start Funding scheme
164022 Platform for Reproducible Acquisition, Processing, and Sharing of Dynamic, Multi-Modal Data 01.07.2016 R'EQUIP

Abstract

The goal of the proposed project SMILE is to pioneer an assessment system for Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) using automatic sign language recognition technology. To achieve this goal, this project uses a multidisciplinary framework that follows two strands of research, one on sign language technology and one on sign assessment with a common link to sign language linguistics. A single institution alone cannot do the very different kinds of research involved in this project. Therefore, a project consortium of three partner institutes with complementary expertise has been built:1. The Idiap Research Institute (Martigny, Switzerland) will coordinate the project and will contribute to the project by developing a novel automatic sign language assessment and feedback approach taking inspirations from a speech recognition approach that was developed under SNSF project FlexASR.2.The Hochschule für Heilpädagogik (HfH in Zurich) will bring its expertise in sign language assessment and sign linguistics (through collaboration with Center for Sign Language Research, FZG, Basel). In addition, the HfH will play a central role in connecting the real world of L2 learners and the deaf community in the German part of Switzerland to the project.3. The University of Surrey (England) will bring to the project its longstanding expertise in sign language technology, visual data acquisition and computer vision and in particular its wide experience in state-of-the-art sign language technology research through European-level projects such as DictaSign.To achieve the end-goals, the project is organized as three sub-projects: 1. Resources and Tools, which will deal with creation of requisite DSGS sign language resources and tools. 2. Sign Language Technology, which will deal with development of an automatic sign assessment system with feedback based on sign language recognition/verification and sign production.3. Assessment of L2 Learners and Feedback, which will develop and standardize a vocabulary test that can be aligned with levels A1 and A2 of the Common European Framework of Reference for Languages (CEFR), and will evaluate automatic sign language assessment system w.r.t human assessment. The SMILE project will involve not only experienced and internationally known researchers in their respective fields, but also young hearing and Deaf team members. The results of the project are expected to have an echo in the larger Deaf community -- not only through the involvement of many Swiss German Deaf signers with the project as subjects, but also because the national Swiss Deaf Association has recently decided to align its sign language curricula to the levels of the Common European Framework of Reference for Languages (CEFR). This SMILE project follows the CEFR approach by developing an assessment system that tests the production of vocabulary of DSGS at level A1 with first time integration of new technologies for sign language. SMILE will thus lay an advanced platform for teaching and learning systems, both specifically for DSGS and as a model for other sign languages.
-