Intelligent Multimedia Interface Agents

Outline of the Tutorial

Multimodal and Multimedia Interfaces

From Multimedia to Intellimedia

Multimedia Access to Information Highways

Limitations of Current Multimedia Technology

AI Meets Multimedia Technology

Guidelines for an Intelligent Multimedia Interface

The Role of Representation Languages in Intelligent Man-Machine Interfaces

Generating and Transforming Presentations in Different Modes and Media

Knowledge-Based Design of Multimedia Dialogs

The Seeheim Model (Pfaff 1985): "Classical" Approach to User Interface Management Systems

What are Agent-based User Interfaces?

The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998)

SmartKom: A Transportable and Transmutable Interface Agent

SmartKom: Intuitive Multimodal Interaction

SmartKom-Public: A Multimodal Com-munication Booth

SmartKom-Mobile: A Handheld Communication Assistant

SmartKom-Home/Office: A Versatile Agent-based Interface

Development of a Standard Reference Model for IMMP-Systems

Components of the Proposed Reference Model

Context and Progress of the Activity

WIP: Adaptive Multimodal Information Presentation in a Distributed Setting (cf. [Wahlster et al. 93])

Combinatorics of Presentation Variants for a Single Presentation Goal

Communicating the Same Information in Multiple Ways

Communicating the Same Information in Multiple Ways Multimodal presentation as illustrated text

Communicating the Same Information in Multiple Ways Variation of the text length

Communicating the Same Information in Multiple Ways

Approaches for Organizing Multimedia Presentations

Generalized Speech-Act Theoretic View: Distinction between Generating and Using a Presentation

Communicative Acts: Rhetorical, Linguistic, and Graphical (cf. Maybury 1993)

Intentional Structure of a Document Fragment (cf. [André/Rist 93])

Rhetorical Structure of a Document Fragment (cf. [André/Rist 93])

What makes Multimedia Generation more Difficult than Text Generation

Schema-Based Approaches in IMP-Systems

Operator-Based Approaches in IMP-Systems

Planning Operator in AIMI (cf. [Maybury 93])

Planning Operator in WIP/PPP (cf. [André/Rist 93])

Criteria for Media Selection

Rhetorical Relations in WIP/PPP: Some Examples

Advantages of the Plan-based Approach to Multimedia Presentation

Architectures for lntelligent Multimedia Systems

SAGE System Architecture (cf. [Roth et al. 91])

COMET System Architecture (cf. [Feiner/McKeown 91])

System Architecture proposed by Arens and Colleages (cf. [Arens et al. 93])

The WIP-Architecture (cf. [Wahlster et al. 91])

Integration of the Presentation Planner into the Multimedia Presentation System WIP

Sequential vs. Interleaved Presentation Design

Hierarchical Blackboard Architecture: (cf. [Han/Zuckerman 95])

Agent-Based Architecture (ct. [Han/Zuckerman 96])

The Open Agent Architecture OAA (cf. [Moran et al. 97])

Overview of OAA

OAA-Based Application: Map-Based Tourist Information (cf. [Cheyer/Julia 95])

Multimodal Map Application

Several Possibilities to Formulate One and the Same Query:

Sample Gestures

Agent Architecture for Map Application

OAA-Based Application Info Wiz: An Animated Voice Interactive Interface to the Web

Interactive Multimedia Presentations

Presentation Styles (cf. [Andre/Rist 94a])

Content Level and Exchange Level

Sample Dialog in PEA (cf. [Moore/Swartout 91])

IDAS Sample Dialog (cf. [Reiter et al. 92])

ALFRESCO: Integrating Hypermedia and NL (cf. [Stock et al. 91])

Multimedia Interface of the FLUIDS Demonstrator

Requirements for User Interfaces in Real-Time Decision Support

The Planning Approach to Intelligent Multimedia Dialog Management

Representing Presentation Knowledge with Presentation Strategies

Projects on Animated Interface Agents

Projects on Animated Interface Agents

Projects on Animated Characters

Commercial Software based on Lifelike Characters (LLCs)

Interjection: Do we really need anthropomorphic interfaces?

Interjection: Do we really get better interfaces by anthropomorphization?

Some Arguments in Favor of Animated Agents

Arguments in Favor of Animated Presentation Agents

Empirical Evaluation of Personalized User Interfaces I

Empirical Evaluation of Personalized User Interfaces II (cf. Lester et al. 97b)

Empirical Evaluation of Personalized User Interfaces III (cf. Takeuchi/Nagao 93)

Relation between Conversational Situations and Facial Displays

Evaluation of the PPP Persona Presentation Agent (cf. Mulken et al. 98)

Experimental Setting for the Evaluation

Questionnaires Used in the Evaluation

Results of the Experiments

Empirical Evaluation of the Persona's Trustworthiness

Empirical Evaluation of the Persona's Trustworthiness

Scenarios: Navigation Support in a Web-like Environment

Procedure

Result (32 Subjects)

Sending Interface Agents to Clients: Plug-Ins or Applets?

The Persona Project at Microsoft Research (cf. Ball et al. 97)

Peedy Indicating a Misconception

Overview of the Persona System

Architecture of Peedy's Animation Control

An Animation Control Example

ReActor Animation Library

The Language of the Animated Controller I

The Language of the Animation Controller II

Algorithm for Assembling Animation Sequences

Summary: Peedy

Deployment of a Presentation Agent in the PPP Context (cf. André et al. 98a)

Deployment of a Presentation Agent in the PPP Context

Deployment of a Presentation Agent in the PPP Context

Deployment of a Presentation Agent in the PPP Context

Determinants of the Persona Behavior

Kinds of Presentation Act

Additional Acts to Enhance Believability

Subtasks of Behavior Planing

Necessary Extensions to the Automated Presentation Generator

Two-Phase Approach to Control the Agents Behavior

Example of a Presentation Strategy

Overview of the Presentation Planning Process in PPP

Two-Phase Approach to Cope with Acts of an Unpredictable Duration

Dynamic Update of the Presentation Schedule at Runtime

Tasks of the Persona Engine

Context-Sensitive Decomposition of Persona Acts

Compilation of Persona Actions

PPP: Summary

AiA: Adaptive Communication Assistant for Effective Infobahn Access (cf. André et al. 98a)

Extensions in AiA

Integration of Additional Web Resources

Which Kind of Information Should the Persona Convey?

How to Get a Useful Representation for Information Found on the WWW?

Possible Distributions of Components

The Hypermedia Model

Navigation Graph for the Hotel Presentation

How to Augment the Presentation Planner with Hypermedia Facilities

Dynamic Node Expansion

AiA: Summary

Commercial Applications:

Herman, the Animated Agent of Design-A-Plant (cf. Lester/Stone 97, Stone/Lester 97)

Sample Application: Cosmo, the Animated Agent of the Internet Advisor System (cf. Lester et al. 97a)

Two Kinds of Behavior: Advisory/Explanatory Behaviors and Believability-Increasing Behaviors

Behavior Sequencing Engine

Sequencing of Advisory and Explanatory Behaviors

Sequencing of Believability-Enhancing Behaviors

Summary: Herman, the Bug

Steve: Integrating Pedagogical Capabilities in a Virtual Environment (cf. Rickel/Johnson 97, Rickel/Johnson 98)

What Steve perceives

Type of Motor Actions

Use of Head-Mounted Displays, 3D-Mouse and Dataglove

Interaction with the Virtual World

Learning Environment with 2 Steve Agents

Menu-Based Interface

Design Criteria for Task Execution

Example of Task Definition

Definition of Tasks

Types of Goals

Demonstrating Task Steps

Monitoring the Student

Generation of Explanations

Summary: Steve

Animated Conversation (cf. Cassell et al. 94)

Example of Symbolic Gesture Specification in Animated Conversation

Communicative Function of Head Movements and Facial Displays

Communicative Function of Hand Gesture

Synchronization of Gesture, Facial Movements and Speech

PaT-Net that Synchronizes Gestures with the Dialog at the Phoneme Level

Summary: Animated Conversation

Swamped!: Using a Plush Toy to Direct Synthetic Characters [Kline/Blumberg 99]

Main Features of Swamped!

Programming Interfaces with Microsoft Agent

Example of programming Microsoft Agent

DFKI PET: Persona-Enabling Toolkit

The Persona Editor

The Persona Markup Language

Persona Active Elements (PAE)

Functional View of PET

Tasks of PET

The Bidirectional Control Flow on Persona-Enabled Webpages

New Projects at DFKI

Multiple Interface Agents

Presentation Teams

Current and Future Work: Multiple Interface Agents for User-adaptive Decision Support

Multi-User Interaction with Multiple Interface Agents for ECommerce on a Single WebTV

PUPPET - Behavior planning for life-like characters in a virtual puppet theatre

PUPPET - Behavior planning for life-like characters in a virtual puppet theatre

Deployment of Life-Like Characters in Virtual Meeting Rooms

Beyond Avatars: Use of life-like characters in the Magic Lounge

Possible Roles for LLC's in the Magic Lounge

What are Net Agents?

Three Generations of Web Sites

What is a Virtual Web Page?

Netbots as Personal Assistants for WWW Users

The Generation of Virtual Webpages with PAN and AiA

The Combination of Retrieved and Generated Media Objects for Virtual Webpages

The Combination of Retrieved and Generated Media Objects for Virtual Webpages

User modeling techniques for WWW applications

User Modelling Techniques for WWW Applications

Agent Residence and Ownership

Softbots as Matchmakers: Electronic Travel Agencies

Intelligent Travel Agent

Retrieval results from the Web are presented using the Netscape NavigatorTM and JavaTM

Retrieved results from the Web are presented in a personalized and data-dependent way

Retrieved material is outlined by abstraction and annotation techniques

The AiA Presentation Server assigns individual AiA processes to each user/client

IKP Design and Integration

Functionality of IKP

The problem: How to find a suitable product (e.g. a car)

Intelligent Interface Technology is a Prerequisite for Advanced WebCommerce

Research on Intelligent Web Services brings disparate subfields in the area of intelligent systems together

Conclusions

Conclusions

Key References: Intelligent User Interfaces

Key References: Animated Interface Agents:

URLS: Intelligent User Interfaces: