Dialog-based Human-Computer Interaction by Coordinated Analysis and Generation of Multiple Modalities

From Spoken Dialogue to Multimodal Dialogue

Merging Various User Interface Paradigms

Multimodal Input and Output in the SmartKom System

Multimodal Interaction with a Life-like Character

SmartKom: Multimodal Dialogs with a Life-like Character

Outline of the Talk

SmartKom: Intuitive Multimodal Interaction

Salient Characteristics of SmartKom

SmartKom: A Transportable Interface Agent

SmartKom-Home on a Portable Webpad

Smartkom-Mobile

SmartKom`s SDDP Interaction Metaphor

Some Complex Behavioural Patterns of the Interaction Agent Smartakus

Visual Support for SDDP

The Perspective of the User

Decomposition of Behavioural Schemata: Phases of Gestures

Some Complex Behavioural Patterns of the Interaction Agent Smartakus

Multimodal Input and Output in SmartKom

Modality-Specific Representation Languages as an Intermediate Representation before Media Fusion

SmartKom‘s Data Collection of Multimodal Dialogs

Mobile Presentation Unit for SmartKom-Public

The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998)

The High-Level Control Flow of SmartKom

Multimodal Access to Telephony Applications via SmartKom

SmartKom: Towards Multimodal and Mobile Dialogue Systems for Indoor and Outdoor Navigation

Getting Driving and Walking Directions via SmartKom

Getting Driving and Walking Directions via SmartKom

Classification of Facial Expressions (U. Erlangen)

Recognizing Neutral Facial Expressions

Recognizing Affect: An Angry Facial Expression of the User

Multimodal Interaction with Consumer Electronic via SmartKom

Combination of Speech and Gesture in SmartKom

Multimodal Input and Output in SmartKom

Three Levels of Mark-up Languages for the Web

M3L Integrates Three Language Families

Mapping Typed Feature Structures onto XML-based M3L Terms

M3L Representation of the Multimodal Discourse Context

M3L Representation of the Word Lattice Produced by the Speech Recognizer for “There [?] I would like to get a reservation.“

Gesture Recognition and Gesture Analysis “There [?] I would like to get a reservation.“

Language Analysis and Media Fusion: Turn8: “There [?] I would like to get a reservation.“

Result of the Action Planner: Presentation Tasks and Presentation Results

Output Synchronization: Speech, Gesture, Graphics, Animation

Language Generation

SmartKom uses a Combination of Concept-to-Speech and Text-to-Speech Technologies

Conclusions