SpeechEval: A Domain-Independent User Simulation Platform for Spoken Dialog System EvaluationTatjana Scheffler; Roland Roller; Norbert Reithinger
In: Ramon Lopez-Cozar Delgado; Tetsunori Kobayashi (Hrsg.). Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop. International Workshop on Spoken Dialogue Systems Technology (IWSDS-11), September 1-3, Granada, Spain, Pages 295-300, ISBN 978-1-4614-1334-9, Springer, 2011.
In this paper, we present the SpeechEval user simulation platform for spoken dialog system (SDS) evaluation. Our user simulation realizes a complete end-to-end user test system with spoken input and output that calls speech-based dialog systems via telephone, and can be used in the development and evaluation of real-life SDSs. Our user simulations are realistic in that they can interact on the speech level, are largely domain and system independent, and can be used with any given spoken dialog system in a black-box manner, with relatively little manual adaptation or training data. They can be used to test SDSs in different stages and to evaluate and compare them, in addition to costly user tests.