DFKI-LT - Dissertation Series

Vol. XXXVI

Hendrik Zender: Situated Production and Understanding of Verbal References to Entities in Large-Scale Space

ISBN: 978-3-933218-35-3
242 pages
price: € 16

order form

The work presented in this thesis addresses the fundamental questions of how machines, such as robots and other autonomous agents, can acquire a mental representation of their environment that allows them to (a) act and navigate in it, and (b) communicate about it with humans in natural language.

We specifically investigate representations of structured environments that cannot be apprehended as a perceptual whole (i.e., large-scale space). This comprises, for instance, indoor domestic environments, or building ensembles. By that, the presented work goes beyond situated natural language interaction about an agent's immediate surroundings (i.e., small-scale space), such as table-tops or single room spaces.

Situated communication about entities that is, things, places, properties, and events in large-scale space requires the interlocutors to draw attention to entities that are not currently observable, and to comprehend which remote places and things are being talked about.

We furthermore show how such representations that can be used for spoken interaction with human users also endow autonomous agents with skills for context-aware planning and execution of actions in structured environments that are made by and for humans. To this end, the presented spatial models have been implemented and deployed in integrated systems for intelligent mobile robots.

We then present an approach for natural language generation and understanding that makes use of the acquired spatial models. It allows an agent to successfully generate and resolve natural language expressions that refer to entities in large-scale space. The approach is backed by observations from an empirical spoken language production experiment. The thesis concludes with a discussion of ongoing work to transfer the models made for intelligent mobile robots to autonomous virtual agents that act in an online virtual 3D world.