Publication
Collecting Subjective Ratings of Voice Likability: Laboratory vs. Crowdsourcing
Rafael Zequeira Jiménez; Laura Fernández Gallardo; Sebastian Möller
In: Workshop on Hybrid Human-Machine Computing (HHMC 2017). Workshop on Hybrid Human-Machine Computing, Guildford, UK, Pages 1-2, 2017.
Abstract
Crowdsourcing has become a powerful approach for rapid collection of user input from a large set of participants at low cost. While previous studies have investigated the acceptability of crowdsourcing for obtaining reliable perceptual scores of audio or video quality, this work examines the suitability of crowdsourcing to collect voice likability ratings. We describe our conducted tests based on direct scaling and on paired-comparisons, that were executed in crowdsourcing using micro-tasks and in the laboratory under controlled conditions. Design considerations are proposed for adapting the laboratory listening tests to a mobile-based crowdsourcing platform to obtain trustworthy listeners’ answers. The likability scores obtained by the different test approaches are highly correlated. This outcome motivates the use of crowdsourcing for future listening tests investigating e.g. speaker characterization, reducing the costs involved in engaging participants and administering the test onsite.