Dynamically Updating and Interrelating Representations of Visual and Linguistic Discourse

John Kelleher, Josef A. van Genabith, Fintan Costello

In: Artificial Intelligence (AIJ) Artificial Intelligence Journal. Volume 67 (1-2). Special issue on Connecting Language to the World. Editors: Deb Roy and Ehud Reiter. 2005. Pages 62-102.. 67 1-2 Seiten 62-102 Elsevier 2005.


The fundamental claim of this paper is that salience—both visual and linguistic—is an important overarching semantic category structuring visually situated discourse. Based on this we argue that computer systems attempting to model the evolving context of a visually situated discourse should integrate models of visual and linguistic salience within their natural language processing (NLP) framework. The paper highlights the importance of dynamically updating and interrelating visual and linguistic discourse context representations. To support our approach, we have developed a real-time, natural language virtual reality (NLVR) system (called LIVE, for Linguistic Interaction with Virtual Environments) that implements an NLP framework based on both visual and linguistic salience. Within this framework saliency information underpins two of the core subtasks of NLP: reference resolution and the generation of referring expressions. We describe the theoretical basis and architecture of the LIVE NLP framework and present extensive evaluation results comparing the system’s performance with that of human participants in a number of experiments.

Weitere Links

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence