DFKI-LT - Using a new analytic measure for the annotation and analysis of MT errors on real data

Arle Richard Lommel, Aljoscha Burchardt, Maja Popovic, Kim Harris, Eleftherios Avramidis, Hans Uszkoreit
Using a new analytic measure for the annotation and analysis of MT errors on real data
5 Proceedings of the 17th Annual Conference of the European Association for Machine Translation, Pages 165-172, Dubrovnik, Croatia, European Association for Machine Translation, Croatian Language Technologies Society, 2014
 
This work presents the new flexible Multidimensional Quality Metrics (MQM) framework and uses it to analyze the performance of state-of-the-art machine translation systems, focusing on “nearly acceptable” translated sentences. A selection of WMT news data and “customer” data provided by language service providers (LSPs) in four language pairs was anno- tated using MQM issue types and examined in terms of the types of errors found in it. Despite criticisms of WMT data by the LSPs, an examination of the resulting errors and patterns for both types of data shows that they are strikingly consistent, with more variation between language pairs and system types than between text types. These results validate the use of WMT data in an analytic approach to assessing quality and show that analytic approaches represent a useful addition to more traditional assessment methodologies such as BLEU or METEOR.
 
Files: BibTeX, Lommel_el_al_2014_MQM.pdf