Class error rates for evaluation of machine translation output

Maja Popovic

In: Proceedings of the Seventh Workshop on Statistical Machine Translation. Workshop on Statistical Machine Translation (WMT-12), 7th, located at NAACL, June 7-8, Montreal, QC, Canada, Pages 71-75, Association for Computational Linguistics, 6/2012.


We investigate the use of error classification results for automatic evaluation of machine translation output. Five basic error classes are taken into account: morphological errors, syntactic (reordering) errors, missing words, extra words and lexical errors. In addition, linear combinations of these categories are investigated. Correlations between the class error rates and human judgments are calculated on the data of the third, fourth, fifth and sixth shared tasks of the Statistical Machine Translation Workshop. Machine translation outputs in five different European languages are used: English, Spanish, French, German and Czech. The results show that the following combinations are the most promising: the sum of all class error rates, the weighted sum optimised for translation into English and the weighted sum optimised for translation from English.


errcats.pdf (pdf, 125 KB )

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence