SpeechEval - Evaluating Spoken Dialog Systems by User Simulation

Tatjana Scheffler; Roland Roller; Norbert Reithinger

In: Arne Jönsson; Jan Alexandersson; David Traum; Ingrid Zukerman (Hrsg.). Proceedings of the 6th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems. IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems (KRPD-09), located at IJCAI 2009, July 12, Pasadena, CA, USA, Pages 93-98, IJCAI 2009, 2009.


In this paper, we introduce the SpeechEval system, a platform for the automatic evaluation of spoken dialog systems on the basis of learned user strategies. The increasing number of spoken dialog systems calls for efficient approaches for their development and testing. The goal of SpeechEval is the minimization of hand-crafted resources to maximize the portability of this evaluation environment across spoken dialog systems and domains. In this paper we discuss the architecture of SpeechEval, as well as the user simulation technique which allows us to learn general user strategies from a new corpus. We present this corpus, the VOICE Awards human-machine dialog corpus, and show how this corpus is used to semi-automatically extract the resources and knowledge bases on which SpeechEval is based.


ijcai2009-finalpaper.pdf (pdf, 137 KB )

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence