Skip to main content Skip to main navigation


A Linguistically Motivated Test Suite to Semi-Automatically Evaluate German--English Machine Translation Output

Vivien Macketanz; Eleftherios Avramidis; Aljoscha Burchardt; He Wang; Renlong Ai; Shushen Manakhimova; Ursula Strohriegel; Sebastian Möller; Hans Uszkoreit
In: Proceedings of the Language Resources and Evaluation Conference. International Conference on Language Resources and Evaluation (LREC), Marseille, France, Pages 936-947, European Language Resources Association, 6/2022.


This paper presents a fine-grained test suite for the language pair German–English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation is carried out with regular expressions. We describe the creation and implementation of the test suite in detail, providing a full list of all categories and phenomena. Furthermore, we present various exemplary applications of our test suite that have been implemented in the past years, like contributions to the Conference of Machine Translation, the usage of the test suite and MT outputs for quality estimation, and the expansion of the test suite to the language pair Portuguese–English. We describe how we tracked the development of the performance of various systems MT systems over the years with the help of the test suite and which categories and phenomena are prone to resulting in MT errors. For the first time, we also make a large part of our test suite publicly available to the research community.


Weitere Links