Situated Production and Understanding of Verbal References to Entities in Large-Scale Space

Hendrik Zender

Saarbrücken Dissertations in Computational Linguistics and Language Technology 36 ISBN 978-3-933218-35-3 German Research Center for Artificial Intelligence and Saarland University Saarbrücken, Germany 12/2011.


The work presented in this thesis addresses the fundamental questions of how machines, such as robots and other autonomous agents, can acquire a mental representation of their environment that allows them to (a) act and navigate in it, and (b) communicate about it with humans in natural language. We specifically investigate representations of structured environments that cannot be apprehended as a perceptual whole (i.e., large-scale space). This comprises, for instance, indoor domestic environments, or building ensembles. By that, the presented work goes beyond situated natural language interaction about an agent's immediate surroundings (i.e., small-scale space), such as table-tops or single room spaces. Situated communication about entities – that is, things, places, properties, and events – in large-scale space requires the interlocutors to draw attention to entities that are not currently observable, and to comprehend which remote places and things are being talked about. We furthermore show how such representations that can be used for spoken interaction with human users also endow autonomous agents with skills for context-aware planning and execution of actions in structured environments that are made by and for humans. To this end, the presented spatial models have been implemented and deployed in integrated systems for intelligent mobile robots. We then present an approach for natural language generation and understanding that makes use of the acquired spatial models. It allows an agent to successfully generate and resolve natural language expressions that refer to entities in large-scale space. The approach is backed by observations from an empirical spoken language production experiment. The thesis concludes with a discussion of ongoing work to transfer the models made for intelligent mobile robots to autonomous virtual agents that act in an online virtual 3D world.


Weitere Links

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence