Call for Workshop Submissions/Participation




Organized by the ACL Special Interest Group

on Multimedia Language Processing (SIGMEDIA)

Madrid, July 11th 1997

(in conjunction with ACL-97/EACL-97)

Workshop Information

What the Workshop is About

A growing number of research projects has started to investigate the use of referring expressions in multimedia systems. On the one hand, the use of multiple media has led to new problems, such as a proper treatment of cross-media references. For example, text may refer to parts of an illustration. On the other hand, it has turned out that many concepts already known from natural language processing, such as cohesion, take on an extended meaning in multimedia discourse. For example, a proper treatment of referring expressions in a multimedia discourse requires an explicit representation of the syntax and semantics of the graphical discourse. As theories of NL reference become more sophisticated, it is quite natural to investigate whether these theories also encompass other media, such as graphics and pointing gestures.

Several research projects have already started to transfer theories to the broader context of multimedia discourse. Examples of models that have been used for multimedia applications are Grosz and Sidner's theory of discourse structure, the centering model developed by Joshi and colleagues and Appelt's and Kronfeld's model of referring. However, there are researchers who doubt that linguistic phenomena, such as anaphora, also exist in multimedia dialogue. The reason they give is that there are no graphical devices for distinguishing between a reference-specifying and a predication-specifying part since objects and their properties are hardly separable once depicted.

The workshop will be centered around questions, such as "To what extent can linguistic models be applied to multimedia references?", "Which linguistic phenomena can also be observed in multimedia discourse?" and "Is a cross-modality theory of reference possible?". Topics of interest include, but are by no means restricted to the following:

Organizing Committee:

Elisabeth André, DFKI, Germany (Email:
Laurent Romary, CRIN-CNRS & INRIA Lorraine, France (Email: )
Thomas Rist, DFKI, Germany (Email:

Programme Committee:

Elisabeth André, DFKI GmbH, Germany
Doug Appelt, SRI International, USA
Jean Caelen, CLIPS-IMAG, France
Robert Dale, Microsoft Research Institute, Australia
John Lee, University of Edinburgh, UK
Luis Pineda, IEE, Mexico
Thomas Rist, DFKI GmbH, Germany
Laurent Romary, CRIN, France
Massimo Zancanaro, IRST, Italy
Bonnie Webber, University of Pennsylvania, USA

Workshop Programme:

Last revised May 16th, 1997
Time schedule subject to change.




9:15 A Syndetic Approach to Referring Phenomena in Multimodal Interaction G.P. Faconti and M. Massink
9:40 A Model for Multimodal Reference Resolution L.A. Pineda and E. G. Garza

Summary and Discussion:

Cross-modality Models and Frameworks

Moderator: Elisabeth André




11:00 Towards Generation of Fluent Referring Action in Multimodal Situations T. Kato and Y.I. Nakano
11:25 Hypertext and Deixis D. Loehr
11:35 Referring in Multimodal Systems: The Importance of User Expertise and System Features D. Petrelli, A. DeAngeli, W. Gerbino and G. Cassano
12:00 Integration and Synchronization of Input Modes during Multimodal Human-Computer Interaction S. Oviatt, A. DeAngeli and K. Kuhn
Presented by: P. Cohen


Common Myths about Multimodal Integration
during Human-Computer Interaction

Moderators: P. Cohen and M. Johnston



15:00 Multimodal References in GEORAL TACTILE J. Siroux, M. Guyomard, F. Multon and C. Rémondeau
15:25 Constraints on the Use of Language, Gesture and Speech for Multimodal Dialogues B. Gaiffe and L. Romary
15:50 Active and Passive Gestures - Problems with the Resolution of Deictic and Elliptic Expressions in a Multimodal System M. Streit
16:00 Scene Direction Based Reference in Drama Scenes H. Nakagawa, Y. Yaginuma and M. Sakauchi

Summary and Discussion
Moderator: Laurent Romary




17:00 Generating Referential Descriptions in Multimodal Environments H. Horacek
17:25 Planning Referential Acts for Animated Presentation Agents E. André and T. Rist
17:50 Exploiting Image Descriptions for the Generation of Referring Expressions K. Hartmann and J. Schöpp
18:00 Referring to Displays in Multimodal Interfaces D. He, G. Ritchie and J. Lee

Summary and Discussion:

Effective Means of Referring

Moderator: Thomas Rist


Final Discussion



Workshop Participation:

Workshop attendance will be limited to maximally 40 people, persons without a submission should contact the organizers as soon as possible. According to the ACL/EACL workshop guidelines, all workshop participants must register for the ACL main conference.

Important Dates:

17th March 1997: (Closed)
Deadline for receipt of submissions
Statements of interest and project descriptions may by submitted by May 12th
4th April 1997: (Done)
Authors informed of acceptance/rejection
24th April 1997: (Closed)
Final versions due for proceedings
31th May 1997:
ACL/EACL-97 Early Registration Deadline
11th July 1997:

Submission Details:

We solicit the submission of full papers or short project descriptions. Full papers should be between five and twelve pages, project descriptions between two and four pages in length. Electronic submissions in postscript format are strongly encouraged and should be emailed to Hard copy (*four* copies of the paper) should be mailed to Thomas Rist (address below).

Preparation of the Camera-Ready Copy:

Authors should send the URL of a postscript file (make sure it is accessible and printable) to Elisabeth André or place their papers in the /pub/incoming directory of DFKI's ftp-Server ( by May 12th 1997. Please note that this is a HARD deadline. If we should not have revised version of your paper by that time, we will have to take the original version. In case you don't intend to make any changes, let Elisabeth Andre know as well. There is no distinction between long and short papers in the proceedings, however, to avoid that we run out of space, you should inform Elisabeth André ASAP if you should intend to extend the original version of your paper by more than 3 pages or if your paper exceeds more than 12 pages. We suggest to follow the ACL formatting guidelines. Please don't use page numbers.

Address for Further Information:

Thomas Rist
German Research Center for Artificial Intelligence
Stuhlsatzenhausweg 3
66123 Saarbrücken
Phone: +49-861/302-5266, Fax: +49-861/302-5341