SmartKom: Dialog-based Human Computer Interaction by Coordinated Analysis and Generation of Multiple Modalities

SmartKom: Merging Various User Interface Paradigms

The SmartKom Consortium

SmartKom‘s Major Scientific Goals

Outline of the Talk

SmartKom Provides Full Symmetric Multimodality

SmartKom Covers the Full Spectrum of Multimodal Discourse Phenomena

SmartKom’s Multimodal Input and Output Devices

Smartkom‘s Control Panel

SmartKom: A Flexible and Adaptive Shell for Multimodal Dialogues

SmartKom`s SDDP Interaction Metaphor

SmartKom‘s Language Model and Lexicon is Augmented on the Fly with Named Entities

The German Federal President E-mailing a Scanned Image with SmartKom’s Help

Interactive Biometric Authentication by Hand Contour Recognition

SmartKom bridges the full loop from multimodal perception to physical action:

Adaptive Perceptual Feedback on the System State

Unification of Scored Hypothesis Graphs for Modality Fusion in SmartKom

M3L Representation of an Intention Lattice Fragment

Fusing Symbolic and Statistical Information in SmartKom

SmartKom‘s Computational Mechanisms for Modality Fusion and Fission

The Markup Language Layer Model of SmartKom

Mapping Digital Content Onto a Variety of Structures and Layouts

The Role of the Semantic Web Language M3L

OIL2XSD: Using XSLT Stylesheets to Convert an OIL Ontology to an XML Schema

Using Ontologies to Extract Information from the Web

M3L as a Meaning Representation Language for the User‘s Input

Exploiting Ontological Knowledge to Understand and Answer the User‘s Queries

SmartKom’s Multimodal Dialogue Back-Bone

A Fragment of a Presentation Goal, as specified in M3L

A Dynamically Generated Multimodal Presentation based on a Presentation Goal

An Excerpt from SmartKom’s Three-Tiered Multimodal Discourse Model

Overlay Operations Using the Discourse Model

The Overlay Operation Versus the Unification Operation

Example for Overlay

Overlay Simulation

Overlay - Scoring

SmartKom‘s Presentation Planner

Adaptive Layout and Plan-Based Animation in SmartKom‘s Multimodal Presentation Generator

Salient Characteristics of SmartKom

The Economic and Scientific Impact of SmartKom

An Example of Technology Transfer: The Virtual Mouse

Former Employees of DFKI and Researchers from the SmartKom Consortium have Founded Five Start-up Companies

SmartKom’s Impact on International Standardization

SmartKom‘s Impact on Software Tools and Resources for Research on Multimodality

Conclusions