Project

Back to overview

A generic platform for CALL based on speech translation

English title A generic platform for CALL based on speech translation
Applicant Bouillon Pierrette
Number 124664
Funding scheme Project funding (Div. I-III)
Research institution ETI / TIM / ISSCO Université de Genève
Institution of higher education University of Geneva - GE
Main discipline Education and learning sciences, subject-specific education
Start/End 01.05.2009 - 30.04.2012
Approved amount 271'939.00
Show all

All Disciplines (2)

Discipline
Education and learning sciences, subject-specific education
Information Technology

Keywords (6)

CALL; speech recognition; machine translation; Computed Aided language Learning; speech translation; translation game

Lay Summary (English)

Lead
Lay summary
It is impossible to acquire any real fluency in a foreign language without being able to practice speaking and listening in an interactive environment, a requirement which often poses serious practical problems. Intuitively, it is natural to consider the idea of developing automatic voice-enabled systems that can play the role of conversation partners. The most obvious way to do this is to construct systems which support free conversation with the student, but experience has shown that robust applications of this kind are difficult and expensive to create, even in limited domains. Recently, however, Wang and Seneff (2007) at MIT have shown that 'spoken translation games' may also be a good way for language students to develop confidence and fluency. The basic strategy is for the machine to prompt the student with a sentence in their own (L1) language, to which the student responds with a spoken translation in the learned (L2) language. Translation, unlike conversational response, can generally be carried out without deep understanding, which implies that 'spoken translation game' systems are comparatively cheap to develop.In this proposal, we will leverage Geneva's Regulus toolkit (Rayner, Hockey and Bouillon, 2006) and other FNRS-sponsored work on speech recognition and translation, to develop a generic CALL platform centered on the 'spoken translation game' idea. We present reasons for believing that the Regulus architecture is very suitable for this type of application. In particular, the platform's grammar-based approach to recognition gives a response profile with accurate recognition on in-grammar utterances and poor or no recognition on out-of-grammar utterances, automatically giving the student feedback on the correctness of their language usage. Regulus's rapid development facilities, based on semi-automatic specialisation of general grammars, also make it easy to create good speech recognisers for domains like travel and tourism, which are of immediate interest to students.All software and resources developed will be distributed free in Open Source form. As with previous projects, we will structure the application so that it can readily be used as a vehicle for doing research in computational linguistics and human language technology.M. Rayner, B.A. Hockey and P. Bouillon, Putting Linguistics into Speech Recognition, CSLI, Stanford, 2006.C. Wang and S. Seneff, Automatic Assessment of Student Translations for Foreign Language Tutoring, in: Proceedings of NAACL/HLT 2007, Rochester, NY", 2007.}
Direct link to Lay Summary Last update: 21.02.2013

Responsible applicant and co-applicants

Employees

Publications

Publication
A Gesture-Controlled Internet-Based Spoken Conversion Partner On A Mobile Device
Tsourakis Nikos (2014), A Gesture-Controlled Internet-Based Spoken Conversion Partner On A Mobile Device, in Universal Access in the Information Society, 13(3), 257-275.
A Textbook-based Serious Game for Practising Spoken Language
Claudia Baur, Emmanuel Rayner, Nikolaos Tsourakis (2013), A Textbook-based Serious Game for Practising Spoken Language, in Proceedings of the 6th International Conference of Education, Research and Innovation (ICERI), SevillaIATED, Sévilla, Spain.
Methodological Issues in Evaluating a Spoken CALL Game: Can Crowdsourcing Help Us Perform Controlled Experiments?
Manny Rayner, Nikolaos Tsourakis (2013), Methodological Issues in Evaluating a Spoken CALL Game: Can Crowdsourcing Help Us Perform Controlled Experiments?, in Proceedings of the Workshop on Speech and Language Technology in Education (SLaTE), GrenobleISCA, France.
A Scalable Architecture For Web Deployment of Spoken Dialogue Systems
Fuchs Matthew, Tsourakis Nikos, Rayner Manny (2012), A Scalable Architecture For Web Deployment of Spoken Dialogue Systems, in Proceedings of LREC, Istanbul, Istanbul, TurkeyEuropean Language Resources Association (ELRA), Paris.
Bootstrapping a Statistical Speech Translator From a Rule-Based One
Rayner Manny, Bouillon Pierrette, Estrella Paula, Nagao Yukie, Christian Gwen (2012), Bootstrapping a Statistical Speech Translator From a Rule-Based One, in Journal of Linguistic Issues in Language Technology, 5, 0-0.
Evaluating Appropriateness Of System Responses In A Spoken CALL Game
Rayner Manny, Bouillon Pierrette, Gerlach Johanna (2012), Evaluating Appropriateness Of System Responses In A Spoken CALL Game, in Proceedings of LREC, Istanbul, TurkeyEuropean Language Resources Association (ELRA), Paris.
Intégration d'un jeu de traduction orale sur le web pour l'apprentissage d'une langue seconde
Pierrette Boiullon, Johanna Gerlach, Claudia Baur, Cristiana Cervini, Racher gasser (2012), Intégration d'un jeu de traduction orale sur le web pour l'apprentissage d'une langue seconde, in Actes du 8ème Colloque Technologies de l’Information et de la Communication pour l’Enseignement, LyonBibliotheques et Archives Canada, Canada.
A Student-Centered Evaluation of a Web-Based Spoken Translation Game
Bouillon Pierrette, Rayner Manny, Tsourakis Nikos, Zhang Q (2011), A Student-Centered Evaluation of a Web-Based Spoken Translation Game, in Proceedings of the SLaTE Workshop, VeniceISCA, France.
Evaluating A Web-Based Spoken Language Translation Game For Learning Domain Language
Bouillon Pierrette, Halimi Sonia, Rayner Manny, Tsourakis Nikos (2011), Evaluating A Web-Based Spoken Language Translation Game For Learning Domain Language, in Proceedings of the International Technology, Education and Development Conference, ValenciaIATED , Spain.
Evaluation of a Mobile Language Learning System Using Language-Neutral Prompts
Tsourakis Nikos, Rayner Manny, Bouillon Pierrette (2011), Evaluation of a Mobile Language Learning System Using Language-Neutral Prompts, in Proceedings of the SLaTE Workshop, veniceISCA, France.
For A Fistful Of Dollars: Using Crowd-Sourcing To Evaluate A Spoken Language CALL Application
Rayner Manny, Frank Ian, Chua Cathy, Tsourakis Nikos, Bouillon Pierrette (2011), For A Fistful Of Dollars: Using Crowd-Sourcing To Evaluate A Spoken Language CALL Application, in Proceedings of the SLaTE Workshop, VeniceISCA, France.
Pour une interlangue utile en traduction automatique de la parole dans des domaines limités
Bouillon Pierrette, Rayner Manny, Estrella Paula, Gerlach Johanna, Georgescul Maria (2011), Pour une interlangue utile en traduction automatique de la parole dans des domaines limités, in Traitement Automatique des Langues (TAL), 52(1), 133-160.
Speech Recognition For Online Language Learning: Connecting CALL-SLT and DALIA
Bouillon Pierrette, Cervini Cristiana, Mandich Anna, Rayner Manny, Tsourakis Nikos (2011), Speech Recognition For Online Language Learning: Connecting CALL-SLT and DALIA, in Proceedings of the International Conference on ICT for Language Learning, Florence, ItalyLibreriaUniversitaria, Italy.
A Multilingual CALL Game Based on Speech Translation
Rayner Manny, Bouillon Pierrette, Tsourakis Nikos, Gerlach Johanna, Georgescul Maria, Nagao Yukie, Baur Claudia (2010), A Multilingual CALL Game Based on Speech Translation, in Proceedings of LREC, ValettaEuropean Language resources Association (ELRA), Paris.
A Multilingual Platform for Building Speech-Enabled Language Courses
Rayner Manny, Bouillon Pierrette, Tsourakis Nikos, Gerlach Johanna, Baur Claudia, Georgecul Maria, Nakao Yukie (2010), A Multilingual Platform for Building Speech-Enabled Language Courses, in Second Language Studies: Acquisition, Learning, Education and Technology, Tokyo, JapanWaseda University, Tokyo, Japan.

Collaboration

Group / person Country
Types of collaboration
CILTA/FACOLTA’ di LINGUE, Université de Bologne Italy (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
Paideia Inc United States of America (North America)
- Industry/business/other use-inspired collaboration
U. Nacional de Córdoba Argentina (South America)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
Future University Hakodate Japan (Asia)
- Publication
Faculty of European Languages and Cultures, Guangdong University of Foreign Studies China (Asia)
- Exchange of personnel

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
Colloque Technologies de l’Information et de la Communication pour l’Enseignement Talk given at a conference Intégration d'un jeu de traduction orale sur le web pour l'apprentissage d'une langue seconde 11.12.2012 Lyon, France Rayner Emmanuel; Bouillon Pierrette;
lrec 2012 Talk given at a conference A Multilingual CALL Game Based on Speech Translation 21.05.2012 Istanbul, Turkey Bouillon Pierrette; Rayner Emmanuel;
ICT2011 Talk given at a conference Speech Recognition For Online Language Learning: Connecting CALL-SLT and DALIA 20.10.2011 Florence, Italy Bouillon Pierrette; Rayner Emmanuel;
INTERACT2011 Talk given at a conference A Gesture-Controlled Internet-Based Spoken Conversion Partner On A Mobile Device 07.09.2011 Lisbon, Portugal, Portugal Rayner Emmanuel; Bouillon Pierrette;
Slate2011 Talk given at a conference A Student-Centered Evaluation of a Web-Based Spoken Translation Game 24.08.2011 Venice, Italy Rayner Emmanuel; Bouillon Pierrette;
IATED2011 Talk given at a conference Proceedings of the International Technology, Education and Development Conference 07.03.2011 Valencia, Spain Bouillon Pierrette; Rayner Emmanuel;
L2WS2010 Talk given at a conference A Multilingual Platform for Building Speech-Enabled Language Courses 22.09.2010 Waseda, Japan Rayner Emmanuel; Bouillon Pierrette;
lrec 2010 Talk given at a conference A Multilingual CALL Game Based on Speech Translation 19.05.2010 Malta, Malta Bouillon Pierrette; Rayner Emmanuel;


Associated projects

Number Title Start Funding scheme
117877 A Swiss Platform for controlled spoken dialog applications 01.01.2008 Project funding (Div. I-III)
153278 Designing and evaluating spoken dialogue based CALL systems 01.05.2014 Project funding (Div. I-III)
109417 Two-way spoken language translator for medical diagnosis 01.10.2005 Project funding (Div. I-III)

Abstract

It is impossible to acquire any real fluency in a foreign language without being able to practice speaking and listening in an interactive environment, a requirement which often poses serious practical problems. Intuitively, it is natural to consider the idea of developing automatic voice-enabled systems that can play the role of conversation partners. The most obvious way to do this is to construct systems which support free conversation with the student, but experience has shown that robust applications of this kind are difficult and expensive to create, even in limited domains. Recently, however, Wang and Seneff at MIT have shown that "spoken translation games" may also be a good way for language students to develop confidence and fluency. The basic strategy is for the machine to prompt the student with a sentence in their own (L1) language, to which the student responds with a spoken translation in the learned (L2) language. Translation, unlike conversational response, can generally be carried out without deep understanding, which implies that "spoken translation game" systems are comparatively cheap to develop.In this proposal, we will leverage Geneva's Regulus toolkit and other FNRS-sponsored work on speech recognition and translation, to develop a generic CALL platform centered on the "spoken translation game" idea. We present reasons for believing that Regulus architecture is very suitable for this type of application. In particular, the platform's grammar-based approach to recognition gives a response profile with accurate recognition on in-grammar utterances and poor or no recognition on out-of-grammar utterances, automatically giving the student feedback on the correctness of their language usage. Regulus's rapid development facilities, based on semi-automatic specialisation of general grammars, also make it easy to create good speech recognisers for domains like travel and tourism, which are of immediate interest to students.The Geneva technology base will give us natural ways to address several major conceptual difficulties in the MIT system. The most serious of these is that the "translation game" involves prompting in the L1, tying acquired L2 words to their L1 counterparts in the student's mind. We will attack this problem by building on methods we have developed for creating human-readable glosses of interlingual representations, and will in particular explore the idea of rendering interlingua in a more language-neutral format. In the interests of improving the student's ability to produce spontaneous speech, we will also experiment with less tightly scripted translation games, giving the student "scenarios" like "buy the items on this shopping list", rather than single sentences to translate. Finally, we will structure the learning game so that students also practice translation from the L2 into the L1. As well as being useful in its own right, this will enable the system to collect a library of recorded native speaker sentences, which we will use as a resource to drive an online spoken help component. All software and resources developed will be distributed free in Open Source form. As with previous projects, we will structure the application so that it can readily be used as a vehicle for doing research in computational linguistics and human language technology.
-