Designing Robust Multimodal Systems for Diverse Users and Mobile Environments

Table of Contents

Designing Robust Multimodal Systems for Diverse Users and Mobile Environments

Introduction to Perceptive Multimodal Interfaces

Advantages of Multimodal Interfaces

The Challenge of Robustness:Unimodal Speech Technology’s Achilles’ Heel

Improved Error Handling in Flexible Multimodal Interfaces

Example of Mutual Disambiguation: QuickSet Interface during Multimodal “PAN” Command

Processing & Architecture

General Research Questions

Study 1- Research Method

Videotape

Study 1- Results

Table 1—Mutual Disambiguation Rates for Native versus Accented Speakers

Table 2- Recognition Rate Differentials between Native and Accented Speakers for Speech, Gesture and Multimodal Commands 

Study 1- Results (cont.)

PPT-Folie

Study 2- Research Analyses

Mobile user with hand-held system & close-talking headset in moderately noisy environment(40-60 dB noise)

Mobile research infrastructure, with user instrumentation and researcher field station

PPT-Folie

Table 3- Mutual Disambiguation Rates during Stationary and Mobile System Use

Table 4- Recognition Rate Differentials during Stationary and Mobile System Use for Speech, Gesture and Multimodal Commands

Study 2- Results (cont.)

PPT-Folie

Future Directions & Challenges

E-Mail: oviatt@cse.ogi.edu

Homepage: www.cse.ogi.edu/CHCC/

download presentation source