A Multimodal Listener Behaviour Driven by Audio Input

Etienne de Sevin, Elisabetta Bevacqua, Sathish Chandra Pammi, Catherine Pelachaud, Marc Schröder, Björn Schuller

In: International Workshop on Interacting with ECAs as Virtual Characters. International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS-10) May 10-14 Toronto Ontario Canada 2010.


Our aim is to build a platform allowing a user to chat with virtual agent. The agent displays audio-visual backchannels as a response to the user's verbal and nonverbal behaviours. Our system takes as inputs the audio-visual signals of the user and outputs synchronously the audio-visual behaviours of the agent. In this paper, we describe the SEMAINE architecture and the data flow that goes from inputs (audio and video) to outputs (voice synthesizer and virtual characters), going through analysers and interpreters. We focus, more particularly, on the multimodal behaviour of the listener model driven by audio input.


aamas2010_desevin.pdf (pdf, 101 KB )

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence