SpeechEval: A Domain-Independent User Simulation Platform for Spoken Dialog System Evaluation

Tatjana Scheffler, Roland Roller, Norbert Reithinger

In: Ramon Lopez-Cozar Delgado , Tetsunori Kobayashi (Hrsg.). Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop. International Workshop On Spoken Dialogue Systems Technology (IWSDS-11) September 1-3 Granada Spain Seiten 295-300 ISBN 978-1-4614-1334-9 Springer 2011.


In this paper, we present the SpeechEval user simulation platform for spoken dialog system (SDS) evaluation. Our user simulation realizes a complete end-to-end user test system with spoken input and output that calls speech-based dialog systems via telephone, and can be used in the development and evaluation of real-life SDSs. Our user simulations are realistic in that they can interact on the speech level, are largely domain and system independent, and can be used with any given spoken dialog system in a black-box manner, with relatively little manual adaptation or training data. They can be used to test SDSs in different stages and to evaluate and compare them, in addition to costly user tests.


Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence