DFKI-LT - Device-Type Influence in Crowd-based Natural Language Translation Tasks (short paper)

Michael Johannes Barz, Neslihan Büyükdemircioglu, Rikhu Prasad Surya, Tim Polzehl, Daniel Sonntag
Device-Type Influence in Crowd-based Natural Language Translation Tasks (short paper)
in: Lora Aroyo, Anca Dumitrache, Praveen Paritosh, Alexander J. Quinn, Chris Welty, Alessandro Checco, Gianluca Demartini, Ujwal Gadiraju, Cristina Sarasua (eds.):
1 Proceedings of the 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing, and Short Paper Proceedings of the 1st Workshop on Disentangling the Relation Between Crowdsourcing and Bias Management (SAD 2018 and CrowdBias 2018) volume 2276,
CEUR Workshop Proceedings, Pages 93-97, Zürich, Switzerland, CEUR-WS.org, 12/2018

The effect of users’ interaction devices and their platform (mobile vs. desktop) should be taken into account when evaluating the performance of translation tasks in crowdsourcing contexts. We investigate the influence of the device type and platform in a crowd-based translation workflow. We implement a crowd translation workflow and use it for translating a subset of the IWSLT parallel corpus from English to Arabic. In addition, we consider machine translations from a state-of-the-art machine translation system which can be used as translation candidates in a human computation workflow. The results of our experiment suggest that users with a mobile device judge translations systematically lower than users with a desktop device, when assessing the quality of machine translations. The perceived quality of shorter sentences is generally higher than the perceived quality of longer sentences.
Files: BibTeX, paper12.pdf, paper12.pdf