Towards Symmetric Multimodality: Fusion and Fission of Speech, Gesture and Facial Expression

SmartKom: Merging Various User Interface Paradigms

The Need for Mobile Multimodal Dialogue Systems

The Fusion of Multimodal Input

The Fusion of Multimodal Input

The SmartKom Consortium

SmartKom‘s Major Scientific Goals

Outline of the Talk

SmartKom Provides Full Symmetric Multimodality

SmartKom Covers the Full Spectrum of Multimodal Discourse Phenomena

SmartKom’s Multimodal Input and Output Devices

SmartKom: A Flexible and Adaptive Shell for Multimodal Dialogues

Generating Maps, Animations and Information Displays on the Fly

Reference Resolution is based on a Symbolic Representation of the Smart Graphics Output

Synchronization of Map Update and Character Behaviour

Interactive Biometric Authentication by Hand Contour Recognition

SmartKom-Home as an Infotainment Companion that helps select media content and runs on a tablet PC

Biometric Authentication, Telephony and Document Scanning and Forwarding in a Multimodal Dialogue

SmartKom-Mobile as a Travel Companion in the Car

SmartKom-Mobile as a Travel Companion for Pedestrians

The High-Level Control Flow of SmartKom

SmartKom‘s Language Model and Lexicon is Augmented on the Fly with Named Entities

Unification of Scored Hypothesis Graphs for Modality Fusion in SmartKom

M3L Representation of an Intention Lattice Fragment

SmartKom Understands Complex Encircling Gestures

Using Facial Expression Recognition for Affective Personalization

Fusing Symbolic and Statistical Information in SmartKom

SmartKom‘s Computational Mechanisms for Modality Fusion and Fission

The Markup Language Layer Model of SmartKom

Spinning the Semantic Web

Mapping Digital Content Onto a Variety of Structures and Layouts

The Role of the Semantic Web Language M3L

OIL2XSD: Using XSLT Stylesheets to Convert an OIL Ontology to an XML Schema

Using Ontologies to Extract Information from the Web

M3L as a Meaning Representation Language for the User‘s Input

Exploiting Ontological Knowledge to Understand and Answer the User‘s Queries

SmartKom’s Multimodal Dialogue Back-Bone

A Fragment of a Presentation Goal, as specified in M3L

A Dynamically Generated Multimodal Presentation based on a Presentation Goal

An Excerpt from SmartKom’s Three-Tiered Multimodal Discourse Model

Overlay Operations Using the Discourse Model

The Overlay Operation Versus the Unification Operation

Example for Overlay

Overlay Simulation

Overlay - Scoring

SmartKom‘s Presentation Planner

Adaptive Layout and Plan-Based Animation in SmartKom‘s Multimodal Presentation Generator

Salient Characteristics of SmartKom

The Economic and Scientific Impact of SmartKom

An Example of Technology Transfer: The Virtual Mouse

Former Employees of DFKI and Researchers from the SmartKom Consortium have Founded Five Start-up Companies

SmartKom’s Impact on International Standardization

SmartKom‘s Impact on Software Tools and Resources for Research on Multimodality

Burning Research Issues in Multimodal Dialogue Systems

Conclusions