SmartKom: Multimodal Dialogs with Mobile Web Users

Merging Various User Interface Paradigms

Code, Media and Modalities

SmartKom: Intuitive Multimodal Interaction

SmartKom: A Transportable and Transmutable Interface Agent

The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998)

SmartKom-Mobile: A Handheld Communication Assistant

SmartKom-Public: A Multimodal Communication Booth

SmartKom-Home/Office: Versatile Agent-based Interface

Integration of Speech and Gesture

XTRA: Interpretation of pointing gestures (eXpert TRAnslator, Wahlster et al. 1986)

Multimodal Input and Output in the SmartKom System

Unification-based Media Fusion

Unification-based Media Fusion

Resource-Sensitive Information Presentation

Augmented Reality: Combining Speech, Gestures and Graphics for Mobile Web Access

Augmented Reality: Combining Speech, Gestures and Graphics for Mobile Web Access

Augmented Reality: Combining Speech, Gestures and Graphics for Mobile Web Access

Multimodal Input and Output in SmartKom

Modality-Specific Representation Languages as an Intermediate Representation before Media Fusion

The SmartKom Control GUI

SmartKom‘s Data Collection of Multimodal Dialogs

ANVIL: Multi-Track Annotation of Video and Language Annotation Tool for Multimodal Interaction

Mobile Presentation Unit for SmartKom-Public

Combination of Speech and Gesture in SmartKom

Three Levels of Mark-up Languages for the Web

M3L Integrates Three Language Families

M3L Representation of the Multimodal Discourse Context

M3L Representation of the Word Lattice Produced by the Speech Recognizer for “There I would like to get a reservation.“

Gesture Recognition and Gesture Analysis “There I would like to get a reservation.“

Language Analysis and Media Fusion: Turn8: “There I would like to get a reservation.“

Result of the Action Planner: Presentation Tasks and Presentation Results

Input into the Language Generator

Language Generation

Output Synchronization: Speech, Gesture, Graphics, Animation