Project

Back to overview

Quality requirements and metrics for the evaluation of machine translation: analysis and integration of expertise

English title Quality requirements and metrics for the evaluation of machine translation: analysis and integration of expertise
Applicant Popescu-Belis Andrei
Number 113604
Funding scheme Project funding (Div. I-III)
Research institution IDIAP Institut de Recherche
Institution of higher education Idiap Research Institute - IDIAP
Main discipline Information Technology
Start/End 01.04.2007 - 31.03.2009
Approved amount 91'120.00
Show all

Keywords (8)

Evaluation of Machine Translation; Quality Models; ISO/IEC Standards; machine translation; evaluation; quality model; quality metrics; ISO 9126

Lay Summary (English)

Lead
Lay summary
Machine translation systems are an important asset of multilingual societies, although it is assumed that they are currently more useful to provide a rough translation of a document or web page than to provide a high-quality translation ready for publication. Such considerations demonstrate the importance of objective measures of the quality of machine translation (MT) systems, and the necessity to relate them to the intended contexts of use of the systems. The metrics for the evaluation of MT systems are however so diverse that it is difficult to choose the right metrics without referring to a framework that compares them qualitatively.

The objective of the present project is to improve our knowledge of the metrics that are used to measure the quality of an MT system in context, and to input into the existing FEMTI tool for evaluation design additional knowledge about the influence of the context of use on the choice of evaluation metrics.

We have previously set up a framework for MT evaluation named FEMTI, which combines a classification of the possible contexts of use of an MT system with a generic quality model for MT that gathers together all possible quality characteristics and evaluation metrics into a unified classification, following recent standards for software evaluation (ISO/IEC 9126 and 14598). FEMTI is web-based resource dedicated to context-based evaluation of MT software, and provides a computer interface that helps evaluators to define a quality model based on evaluation requirements. The tool is implemented thanks to a dynamic document server using an XML representation of all context and quality characteristics (see http://www.issco.unige.ch/femti).

The project is a continuation of a previous SNSF project (see http://www.issco.unige.ch/projects/mteval-fnrs-2004-2006.html). Its main goal is to integrate into FEMTI the existing expertise regarding the definition of quality models for MT and the selection of quality metrics.
The work covers three main areas: (1) case-study analysis of MT evaluation metrics in terms of correlation and scales; (2) input of expertise into FEMTI's core correspondence (the generic contextual quality model), thanks to appropriate interfaces to be used by experts; and (3) documentation of FEMTI for users, experts and maintainers. This will increase the added value of the FEMTI tool, by comparison with a simple repertoire of qualities and metrics, and will ensure its continued use by MT evaluators from various backgrounds.
Direct link to Lay Summary Last update: 21.02.2013

Responsible applicant and co-applicants

Employees

Associated projects

Number Title Start Funding scheme
103318 Quality models and resources for the evaluation of machine translation 01.10.2004 Project funding (Div. I-III)

-