DFKI-LT - Evaluating Machine Translation in a Usage Scenario

Rosa Gaudio, Aljoscha Burchardt, António Branco
Evaluating Machine Translation in a Usage Scenario
in: Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis (eds.):
3 Proceedings of the Tenth International Conference on Language Resources and Evaluation, Portoroz, Slovenia, European Language Resources Association (ELRA), Paris, France, 5/2016
 
In this document we report on a user-scenario-based evaluation aiming at assessing the performance of machine translation (MT) systems in a real context of use. We describe a sequel of experiments that has been performed to estimate the usefulness of MT and to test if improvements of MT technology lead to better performance in the usage scenario. One goal is to find the best methodology for evaluating the eventual benefit of a machine translation system in an application. The evaluation is based on the QTLeap corpus, a novel multilingual language resource that was collected through a real-life support service via chat. It is composed of naturally occurring utterances produced by users while interacting with a human technician providing answers. The corpus is available in eight different languages: Basque, Bulgarian, Czech, Dutch, English, German, Portuguese and Spanish.
 
Files: BibTeX, 58_Paper.pdf