Assessing Inter-Annotator Agreement for Translation Error Annotation

Arle Richard Lommel; Maja Popovic; Aljoscha Burchardt

In: MTE: Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. International Conference on Language Resources and Evaluation (LREC-14), located at LREC 14, May 26-31, Reykjavik, Iceland, LREC, 2014.


One of the key requirements for demonstrating the validity and reliability of an assessment method is that annotators are able to apply it consistently. This paper explores inter-annotator agreement for error classification task and investigates some of the facts that contribute to low IAA.


LREC-Lommel-Burchardt-Popovic.pdf (pdf, 525 KB )

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence