Ehud Reiter

Choosing a Media for Presenting Information

[Full Text]
[msg to moderator]
[debate procedure]

Overview of interactions

No Comment(s) Answer(s) Continued discussion
1 26.11.97 Lewis Johnson
15.12.97 Ehud Reiter

2 14.12.97 Elisabeth André
15.12.97 Ehud Reiter

3 29.1.98 Niels Ole Bernsen

C1. Lewis Johnson (26.11):

This is a good and timely issue to raise for discussion, and thanks to Ehud for contributing it.

Ehud raises a number of points, every one of which deserves careful investigation. I can't do them all justice in this brief reply. But here are a couple of comments:

Some of the problems that Ehud mentions about graphics may be true for pure graphical depictions, but are easily corrected in hybrid and complex presentations. Graphical presentations can be augmented in any of a number of ways, e.g. using legends, textual callouts, highlighting, animation sequences. Many of the limitations that Ehud cites for graphical representations can be remedied by these means. I agree that the presentation author must be aware of these limitations, in order to compensate for them.

Readers of graphical presentations must interpret not just the meaning of individual symbols, but also the meaning to be assigned to spatial layout within the presentation. The reader of the presentation needs to know what dimensions are important, and what units of measurement to apply to a given dimension. The problems alluded to in the Marks and Reiter paper sound like misinterpretations of spatial layout. Spatial layout is particularly susceptible to misinterpretation, because it has to do with the meaning of relationships between symbols, instead of meaning of individual symbols.

Ehud says that textual information is easier to memorize than graphical information. In evaluating this claim I think we need to make a distinction between graphical presentations of unfamiliar information and graphical presentations that overlay unfamilar information on top of familar information. The most common examples are the latter presentations that overlay information onto a geographic map. My favorite example is the route map that appears in the back of most airline magazines, in which a network of routes is overlayed on top of a map of the world. This presentation is much better to learn from than either a textual presentation of a list of routes or an unordered network of nodes and arcs. The cartographic background provides the viewer a framework for organizing what would otherwise be a complex jumble of information, a framework that facilitates both understanding and memory.


Lewis Johnson

W. Lewis Johnson WWW:
Project Leader, Educational Technology Group Tel: (310) 822-1511
Research Assoc. Professor in Computer Science Fax: (310) 823-6714
USC/Information Sciences Institute
4676 Admiralty Way, Suite 1001
Marina del Rey, CA 90292-6695

A1. Ehud Reiter to Lewis Johnson (15.12.97):

Many thanks to Lewis Johnson for his response to my discussion note. I agree completely that many of the problems with pure graphical depictions can be resolved by moving to hybrid presentations that include both text and graphics.

With regard to spatial layout, the problem I was alluding to was that users often make inferences from layout that go beyond the formal notation used in the graphic. I think the best example of this is Petre's flip-flop. If an experienced engineer sees a circuit diagram that contains two NAND gates which are close together, vertically aligned, and have some interconnection, then he may assume that these NAND gates form a bistable flip-flop without actually checking the wiring to verify this. Therefore, a computer system which produces circuit diagrams must be careful not to accidentally put two NAND gates in this configuration when they do not actually constitute a flip-flop. And this must be done not just for flip-flops, but for all spatial configurations which are recognised as "gestalts" by users. This adds considerable complexity to the task of automatically generating such diagrams.

Lewis also suggests we look at maps as a particular type of graphic, and I think this is a good idea, as it makes the discussion more concrete. Maps are interesting because their primary purpose is usually to communicate geographical location, which is something that graphics does very well. Furthermore, road maps in particular are a well-established genre whose conventions are understood by the general public. For example, road widths are usually not drawn to scale on road maps, but users realise this, so they will not incorrectly assume that a road drawn with a 1mm thick line on a 1:100000 scale map must have a physical width of 100m.

The usefullness of airline route maps, which Lewis mentions in his note, depend on the task they are used for. I don't think they are particularly well-suited to helping people memorise an airline's routes; but then this is not something that many people want to do. I would also argue, though, that they are poorly suited to helping people plan trips, which is a very common task; certainly I personally use timetables instead of route maps for this task. This is because

This is not to say that route maps are useless. I suspect that from the airline's perspective, they are marketing tools, whose purpose is to spark people's interest in taking holidays in far-off places, and to reinforce a perception that a particular airline is "big". I would not be at all surprised if they were reasonably effective from this perspective.

C2. Elisabeth André (14.12.97):

Ehud - Thanks very much for initiating the discussion. I have the following comments:

You write that graphics is better suited to conveying information about individual objects than to communicating information about classes of objects. On the other hand, illustrations are often used to enable the reader to classify objects, e.g., as mushrooms or toadstools. I think the point is that graphics often don't make clear whether they stand for an individual or represent a class of individuals. And even if this should be clear, the reader still has to find out whether, for example, a picture of a sheepdog stands for a sheepdog, a dog, a mammal, an animal etc.

This is related to one of the other problems you mention in your paper, namely that graphical presentations often don't make clear what information is intentionally being communicated. My favorite example goes back to Wittgenstein. A picture of a boxer may be used to show what a good posture of a boxer is, what a bad posture of a boxer is, what posture a particular boxer took on in a particular situation etc. (translated from German).

Another important factor you discuss is the communicative purpose of the information. As examples you mention analytical tasks and marketing tasks. I think we should distinguish here between the task the user has to accomplish (e.g., to recognize an increase in car production by studying a bar chart) and the communicative function of a document part (e.g., to provide a motivation for a request). To investigate the influence of the communicative function, we found it very useful to start from the set of rhetorical relations set up by Mann and Thompson. It turned out that a number of these relations can be effectively conveyed via graphical means (cf. [André/Rist 93]). Just some examples:

Furthermore, these examples illustrate that it also depends on the document context whether a picture is more appropriate than text. For example, the media for two object descriptions should not be chosen independently of each other if the objects are to be contrasted with each other.



André, E. and Rist, T.: The Design of Illustrated Documents as a Planning Task. In: M. Maybury: Intelligent Multimedia Interfaces, AAAI Press, pp. 94-116, 1993.

A2. Ehud Reiter to Elisabeth André (15.12.97):

Elizabeth - Hello, thanks very much for your note. I'm glad to see that we agree that one of the problems with using graphics is lack of "precision" (for want of a better word); it is often difficult for users to determine precisely what information a graphic is intended to communicate.

With regard to your discussion on using rhetorical relations to design graphics, I think this brings up the fascinating cognitive-science question of what similarities exist between text and graphics. One similarity is, as you say, the fact that rhetorical relations derived from studying texts also seem to apply to graphics. Other similarities include the importance of sublanguage or genre in both text and graphics [Reiter 1995], and the impact of conversational implicature on both text and graphics [Marks and Reiter 1990]. In these cases, as in your example, it seems that theories originally developed for text also apply to graphics. No doubt there are also examples of theories developed for graphics which also apply to text.

Perhaps, being somewhat speculative, these theories (rhetorical relations, sublanguage/genre, conversational implicature) in fact apply to all forms of human communication, regardless of genre. That is, they are facts about the manner in which people communicate, not facts about language or graphics per se. If this is indeed the case, I think this would be a very exciting finding indeed.



Marks, J. and Reiter, E. (1990). Avoiding Unwanted Conversational Implicatures in Text and Graphics. Proceedings of AAAI-1990, vol 1, pages 450-456.

Reiter, E (1995). Sublanguages in Text and Graphics. Proceedings of IMMI-1995.

C3. Niels Ole Bernsen (29.1.98):


Ehud, thanks for opening an ETAI discussion on a subject which have been a preoccupation of mine for some time. If you don't mind, I have chosen to attach below a paper which addresses several of the claims you made from an attempted "deep analysis" of analogue graphics and natural language. The analysis is part of an undertaking called "Modality Theory" which I will not bother you with at this time except to say that it (1) includes sub-divisions of both "natural language" and " analogue graphics" into several unimodal forms of representation of information, which seems necessary and was remarked several times in the ongoing discussion by Lewis and Elisabeth, and (2) points out that "graphics" and "language" is a false opposition insofar as natural language text is itself a form of graphics. For that reason, "graphics" in Modality Theory is not a `modality' but a `medium', just as acoustics and haptics are media, not modalities. Modality Theory addresses your point 2, "the type of information being communicated", i.e. the different forms of expression of information in the media of graphics, acoustics and haptics. When applied to practical interface design in what we call Information Mapping, i.e. the theoretical support of modality choice for specific applications, one has to include consideration of many other factors, among which those you mention, such as pragmatics, user background and manner of use.

Just to briefly relate your claims to the paper below (replacing your `graphics' with `analogue graphics'):

- physical vs. abstract information: this claim is scoped and justified in the paper below with reference to the distinction between analogue and non-analogue representation;

- diverse information vs. related facts: this claim may be too general. For instance, logical relations, being abstract, are better represented through language;

- information on individual objects vs. information on classes of objects: this claim is scoped and justified in the paper below with reference to the distinction between analogue and non-analogue representation. Elisabeth's point about graphics being used to represent classes of objects, and the ensuing ambiguity, is addressed in the paper below through the notion of the `interpretational scope' of analogue graphics.

Lewis Johnson points out the advantages of using combinations of unimodal modalities for presenting information rather than relying on unimodal representations. This claim is strongly supported by the paper which presents the advantages of combining analogue graphics and language into multimodal representations of information. Lewis also suggests that maps constitute a particular type of representation. Modality Theory makes that point. In Modality Theory, tables are not modalities but `modality structures' having particular useful properties. The novice/expert problem wrt. understanding particular representations is addressed in Modality Theory through the notions of `notation' and `arbitrary representations' where users cannot rely on generally known systems of meaning.

The standard presentation of modality theory is: Niels Ole Bernsen: A Toolbox of Output Modalities. It can be found in html at URL:



Niels Ole Bernsen WWW:
The Maersk Mc-Kinney Moller Institute for Production Technology Tel: (+45) 65 57 35 51
Odense University Fax: (+45) 65 57 35 55
Campusvej 55
5230 Odense M

Niels Ole Bernsen
Why are Analogue Graphics and Natural Language both Needed in HCI?

[Published as: Bernsen, N.O.: Why are Analogue Graphics and Natural Language both Needed in HCI? In F. Paterno (Ed.): Interactive Systems: Design, Specification, and Verification. Focus on Computer Graphics. Springer Verlag 1995, 235-51. Copyright Springer Verlag 1995.]

Abstract:The combined use of language and analogue graphics for the expression of information probably is as old as language itself. The paper addresses the question why we need both the expressions of natural language and analogue graphics for the representation of information. It is argued that analogue graphics and natural language have the complementary expressive virtues of specificity and focus, respectively. Their corresponding lack of focus and specificity, respectively, explain why (a) both have developed a number of mechanisms for coping with these deficiencies and (b) why their combination may often have superior expressive power. Since specificity follows from the analogue character of analogue graphics rather than from their graphic character, analogue sound and touch representations are analysed to explore whether results from the analysis of analogue graphics and their complementarity with natural language can be transferred to other analogue modalities of expression. The paper exemplifies the comparatively new field of Modality Theory.

Full text: here

Additional questions and answers will be added here.
To contribute, please click [send contribution] above and send your question or comment as an E-mail message.
For additional details, please click [debate procedure] above.
This debate is moderated by Elisabeth André.