Skip to main content Skip to main navigation


A Unified Approach for Semantic-Based Multimodal Interaction

Markus Löckelt; Matthieu Deru; Christian H. Schulz; Simon Bergweiler; Tilman Becker; Norbert Reithinger
In: Wolfgang Wahlster; Hans-Joachim Grallert; Stefan Wess; Hermann Friedrich; Thomas Widenka (Hrsg.). Towards the Internet of Services: The THESEUS Research Program. Pages 131-144, Cognitive Technologies, ISBN 978-3-319-06754-4, Springer, 10/2014.


This chapter gives an overview of the basic dialogue shell for multimodal interaction that was created as the foundation for the software components developed in the sub-tasks in the CTC work package 4 of the Theseus research programme. It explains the basic setup of WP4 systems for specific use cases. The general processing paradigm for interaction in our framework using Semantic Interface Elements as well as several dialogue phenomena handled by the dialogue shell are described. Finally, a selection of different systems that were built for a range of applications is presented briefly. They offer access to quite varied use-case scenarios featuring multimodal input and output, including gestures, multi-touch, spoken input and output, and interactions across mobile and stationary devices.