Table of ContentsTowards Symmetric Multimodality: Fusion and Fission of Speech, Gesture and Facial Expression SmartKom: Merging Various User Interface Paradigms The Need for Mobile Multimodal Dialogue Systems The Fusion of Multimodal Input The Fusion of Multimodal Input The SmartKom Consortium SmartKom‘s Major Scientific Goals Outline of the Talk SmartKom Provides Full Symmetric Multimodality SmartKom Covers the Full Spectrum of Multimodal Discourse Phenomena SmartKom’s Multimodal Input and Output Devices SmartKom: A Flexible and Adaptive Shell for Multimodal Dialogues Generating Maps, Animations and Information Displays on the Fly Reference Resolution is based on a Symbolic Representation of the Smart Graphics Output Synchronization of Map Update and Character Behaviour Interactive Biometric Authentication by Hand Contour Recognition SmartKom-Home as an Infotainment Companion that helps select media content and runs on a tablet PC Biometric Authentication, Telephony and Document Scanning and Forwarding in a Multimodal Dialogue SmartKom-Mobile as a Travel Companion in the Car SmartKom-Mobile as a Travel Companion for Pedestrians The High-Level Control Flow of SmartKom SmartKom‘s Language Model and Lexicon is Augmented on the Fly with Named Entities Unification of Scored Hypothesis Graphs for Modality Fusion in SmartKom M3L Representation of an Intention Lattice Fragment SmartKom Understands Complex Encircling Gestures Using Facial Expression Recognition for Affective Personalization Fusing Symbolic and Statistical Information in SmartKom SmartKom‘s Computational Mechanisms for Modality Fusion and Fission The Markup Language Layer Model of SmartKom Spinning the Semantic Web Mapping Digital Content Onto a Variety of Structures and Layouts The Role of the Semantic Web Language M3L OIL2XSD: Using XSLT Stylesheets to Convert an OIL Ontology to an XML Schema Using Ontologies to Extract Information from the Web M3L as a Meaning Representation Language for the User‘s Input Exploiting Ontological Knowledge to Understand and Answer the User‘s Queries SmartKom’s Multimodal Dialogue Back-Bone A Fragment of a Presentation Goal, as specified in M3L A Dynamically Generated Multimodal Presentation based on a Presentation Goal An Excerpt from SmartKom’s Three-Tiered Multimodal Discourse Model Overlay Operations Using the Discourse Model The Overlay Operation Versus the Unification Operation Example for Overlay Overlay Simulation Overlay - Scoring SmartKom‘s Presentation Planner Adaptive Layout and Plan-Based Animation in SmartKom‘s Multimodal Presentation Generator Salient Characteristics of SmartKom The Economic and Scientific Impact of SmartKom An Example of Technology Transfer: The Virtual Mouse Former Employees of DFKI and Researchers from the SmartKom Consortium have Founded Five Start-up Companies SmartKom’s Impact on International Standardization SmartKom‘s Impact on Software Tools and Resources for Research on Multimodality Burning Research Issues in Multimodal Dialogue Systems Conclusions |
Author:Wolfgang Wahlster
E-Mail: wahlster@dfki.de Homepage: http://www.dfki.de/~wahlster |