Software Platforms and Toolkits for Building Multimodal Systems and Applications

Michael Feld; Robert Neßelrath; Tim Schwartz

In: Sharon Oviatt; Björn Schuller; Philip R. Cohen; Daniel Sonntag; Gerasimos Potamianos; Antonio Krüger (Hrsg.). The Handbook of Multimodal-Multisensor Interfaces, Volume 3 -- Language Processing, Software, Commercialization, and Emerging Directions. Chapter 4, Software Platforms and Toolkits for Building Multimodal Systems and Applications, Pages 145-190, ACM Books series (ACM Books), Vol. #23, ISBN 978-1-97000-175-4, Morgan & Claypool Publishers, 7/2019.


This chapter introduces various concepts needed for the realization of multimodal systems. Alongside an overview of the evolution of multimodal dialogue platform architectures, we give an overview of the major components found in most of today’s architectures: input and output processing; fusion and discourse processing; dialogue management; fission and presentation planning; and middleware. We compare several different dialogue management approaches, look in more detail at how the fusion component works, and introduce dialogue act annotation with communicative functions. We will explain the multimodal reference resolution process and consider the special case of cross-modal references. Finally, we present SiAM- dp, an actual multimodal dialogue platform used in a number of research projects and prototypes and highlight some of its particular features.


Weitere Links

German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz