SmartKom: Fusion and Fission of Speech, Gestures, and Facial Expressions

28.11.02


click here to start


Table of Contents

SmartKom: Fusion and Fission of Speech, Gestures, and Facial Expressions

SmartKom: Merging Various User Interface Paradigms

Symbolic and Subsymbolic Fusion of Multiple Modes

Outline of the Talk

SmartKom: A Highly Portable Multimodal Dialogue System

SmartKom: Intuitive Multimodal Interaction

SmartKom`s SDDP Interaction Metaphor

Multimodal Input and Output in the SmartKom System

Symbiotic Interaction with a Life-like Character

Multimodal Input and Output in SmartKom Fusion and Fission of Multiple Modalities

SmartKom‘s Data Collection of Multimodal Dialogs

Personalized Interaction with WebTVs via SmartKom (DFKI with Sony, Philips, Siemens)

Using Facial Expression Recognition for Affective Personalization

Recognizing Affect: A Negative Facial Expression of the User

The SmartKom Demonstrator System

Combination of Speech and Gesture in SmartKom

Multimodal Input and Output in SmartKom

Getting Driving and Walking Directions via SmartKom

Getting Driving and Walking Directions via SmartKom

SmartKom: Multimodal Dialogues with a Hybrid Navigation System

Salient Characteristics of SmartKom

The High-Level Control Flow of SmartKom

SmartKom’s Multimodal Dialogue Back-Bone

Unification of Scored Hypothesis Graphs for Modality Fusion in SmartKom

SmartKom‘s Computational Mechanisms for Modality Fusion and Fission

The Overlay Operation Versus the Unification Operation

Overlay Operations Using the Discourse Model

An Example of the Overlay Operation

Smartkom‘s Three-Tiered Discourse Model

The High-Level Control Flow of SmartKom

Smartakus is a Self-Animated Interface Agent

Some Complex Behavioural Patterns of the Interaction Agent Smartakus

M3L Representation of the Multimodal Discourse Context

M3L Specification of a Presentation Task

SmartKom‘s Presentation Planner

SmartKom‘s Use of Semantic Web Technology

Conclusions

First International Conference on Perceptive & Multimodal User Interfaces (PMUI’03)

Author:Wolfgang Wahlster

E-Mail: wahlster@dfki.de

Homepage: http://www.dfki.de/~wahlster

Download Presentation Source