Ehud Reiter Choosing a Media for Presenting Information

[Full Text] [msg to moderator] [debate procedure] [copyright]

C1. Lewis Johnson (26.11):

This is a good and timely issue to raise for discussion, and thanks to Ehud for contributing it.

Ehud raises a number of points, every one of which deserves careful investigation. I can't do them all justice in this brief reply. But here are a couple of comments:

Some of the problems that Ehud mentions about graphics may be true for pure graphical depictions, but are easily corrected in hybrid and complex presentations. Graphical presentations can be augmented in any of a number of ways, e.g. using legends, textual callouts, highlighting, animation sequences. Many of the limitations that Ehud cites for graphical representations can be remedied by these means. I agree that the presentation author must be aware of these limitations, in order to compensate for them.

Readers of graphical presentations must interpret not just the meaning of individual symbols, but also the meaning to be assigned to spatial layout within the presentation. The reader of the presentation needs to know what dimensions are important, and what units of measurement to apply to a given dimension. The problems alluded to in the Marks and Reiter paper sound like misinterpretations of spatial layout. Spatial layout is particularly susceptible to misinterpretation, because it has to do with the meaning of relationships between symbols, instead of meaning of individual symbols.

Ehud says that textual information is easier to memorize than graphical information. In evaluating this claim I think we need to make a distinction between graphical presentations of unfamiliar information and graphical presentations that overlay unfamilar information on top of familar information. The most common examples are the latter presentations that overlay information onto a geographic map. My favorite example is the route map that appears in the back of most airline magazines, in which a network of routes is overlayed on top of a map of the world. This presentation is much better to learn from than either a textual presentation of a list of routes or an unordered network of nodes and arcs. The cartographic background provides the viewer a framework for organizing what would otherwise be a complex jumble of information, a framework that facilitates both understanding and memory.

Cheers,

Lewis Johnson

A1. Ehud Reiter to Lewis Johnson (15.12.97):

Many thanks to Lewis Johnson for his response to my discussion note. I agree completely that many of the problems with pure graphical depictions can be resolved by moving to hybrid presentations that include both text and graphics.

With regard to spatial layout, the problem I was alluding to was that users often make inferences from layout that go beyond the formal notation used in the graphic. I think the best example of this is Petre's flip-flop. If an experienced engineer sees a circuit diagram that contains two NAND gates which are close together, vertically aligned, and have some interconnection, then he may assume that these NAND gates form a bistable flip-flop without actually checking the wiring to verify this. Therefore, a computer system which produces circuit diagrams must be careful not to accidentally put two NAND gates in this configuration when they do not actually constitute a flip-flop. And this must be done not just for flip-flops, but for all spatial configurations which are recognised as "gestalts" by users. This adds considerable complexity to the task of automatically generating such diagrams.

Lewis also suggests we look at maps as a particular type of graphic, and I think this is a good idea, as it makes the discussion more concrete. Maps are interesting because their primary purpose is usually to communicate geographical location, which is something that graphics does very well. Furthermore, road maps in particular are a well-established genre whose conventions are understood by the general public. For example, road widths are usually not drawn to scale on road maps, but users realise this, so they will not incorrectly assume that a road drawn with a 1mm thick line on a 1:100000 scale map must have a physical width of 100m.

The usefullness of airline route maps, which Lewis mentions in his note, depend on the task they are used for. I don't think they are particularly well-suited to helping people memorise an airline's routes; but then this is not something that many people want to do. I would also argue, though, that they are poorly suited to helping people plan trips, which is a very common task; certainly I personally use timetables instead of route maps for this task. This is because

• Information: Route maps tell me which cities are connected, but not how frequent flights are on a route, or even for that matter whether it is possible to fly a particular segment (eg, even if the United Airlines route map shows planes travelling from Sydney to Melbourne, United cannot sell Sydney->Melbourne tickets for political reasons). I think it would be difficult to convey schedule, frequency, and segment constraints on route maps; in contrast, it is very easy to communicate this information in a timetable.
• Conventions: On a number of occasions I have looked at route maps and said "gosh, I didn't know that airline X flew to city Y" - only to discover, by checking the airline's timetable, that they did not in fact have regular flights to city Y. Some airlines show only currently operating scheduled flights on their map, some also show regular charter flights, some also show cargo routes, and some show all routes which they are authorized to operate, even if they are not currently flying passengers on some of them. In other words, there is no standard set of conventions for what information is displayed in a route map; and this can mislead users who do not take the time to read the fine print. In contrast, there is a reasonable amount of standardisation for timetables, and I personally do not recall ever having been "misled" by a timetable in the same way I was "misled" by some route maps.

C2. Elisabeth André (14.12.97):

Ehud - Thanks very much for initiating the discussion. I have the following comments:

You write that graphics is better suited to conveying information about individual objects than to communicating information about classes of objects. On the other hand, illustrations are often used to enable the reader to classify objects, e.g., as mushrooms or toadstools. I think the point is that graphics often don't make clear whether they stand for an individual or represent a class of individuals. And even if this should be clear, the reader still has to find out whether, for example, a picture of a sheepdog stands for a sheepdog, a dog, a mammal, an animal etc.

This is related to one of the other problems you mention in your paper, namely that graphical presentations often don't make clear what information is intentionally being communicated. My favorite example goes back to Wittgenstein. A picture of a boxer may be used to show what a good posture of a boxer is, what a bad posture of a boxer is, what posture a particular boxer took on in a particular situation etc. (translated from German).

Another important factor you discuss is the communicative purpose of the information. As examples you mention analytical tasks and marketing tasks. I think we should distinguish here between the task the user has to accomplish (e.g., to recognize an increase in car production by studying a bar chart) and the communicative function of a document part (e.g., to provide a motivation for a request). To investigate the influence of the communicative function, we found it very useful to start from the set of rhetorical relations set up by Mann and Thompson. It turned out that a number of these relations can be effectively conveyed via graphical means (cf. [André/Rist 93]). Just some examples:

• Evidence: Since pictures increase authenticity, they are well suited to support a verbal claim.
• Contrast: To emphasize differences or parallels between two objects, the same medium should be chosen for describing both of them.
• Motivate: Pictures may motivate the user to comply with a verbal request. Think of an advertisement showing a cup of steaming coffee to motivate people to buy this coffee.

Best,

Elisabeth

References:
André, E. and Rist, T.: The Design of Illustrated Documents as a Planning Task. In: M. Maybury: Intelligent Multimedia Interfaces, AAAI Press, pp. 94-116, 1993.

A2. Ehud Reiter to Elisabeth André (15.12.97):

Elizabeth - Hello, thanks very much for your note. I'm glad to see that we agree that one of the problems with using graphics is lack of "precision" (for want of a better word); it is often difficult for users to determine precisely what information a graphic is intended to communicate.

With regard to your discussion on using rhetorical relations to design graphics, I think this brings up the fascinating cognitive-science question of what similarities exist between text and graphics. One similarity is, as you say, the fact that rhetorical relations derived from studying texts also seem to apply to graphics. Other similarities include the importance of sublanguage or genre in both text and graphics [Reiter 1995], and the impact of conversational implicature on both text and graphics [Marks and Reiter 1990]. In these cases, as in your example, it seems that theories originally developed for text also apply to graphics. No doubt there are also examples of theories developed for graphics which also apply to text.

Perhaps, being somewhat speculative, these theories (rhetorical relations, sublanguage/genre, conversational implicature) in fact apply to all forms of human communication, regardless of genre. That is, they are facts about the manner in which people communicate, not facts about language or graphics per se. If this is indeed the case, I think this would be a very exciting finding indeed.

Best

Ehud

References:
Marks, J. and Reiter, E. (1990). Avoiding Unwanted Conversational Implicatures in Text and Graphics. Proceedings of AAAI-1990, vol 1, pages 450-456.

Reiter, E (1995). Sublanguages in Text and Graphics. Proceedings of IMMI-1995.

C3. Niels Ole Bernsen (29.1.98):

Ehud, thanks for opening an ETAI discussion on a subject which have been a preoccupation of mine for some time. If you don't mind, I have chosen to attach below a paper which addresses several of the claims you made from an attempted "deep analysis" of analogue graphics and natural language. The analysis is part of an undertaking called "Modality Theory" which I will not bother you with at this time except to say that it (1) includes sub-divisions of both "natural language" and " analogue graphics" into several unimodal forms of representation of information, which seems necessary and was remarked several times in the ongoing discussion by Lewis and Elisabeth, and (2) points out that "graphics" and "language" is a false opposition insofar as natural language text is itself a form of graphics. For that reason, "graphics" in Modality Theory is not a `modality' but a `medium', just as acoustics and haptics are media, not modalities. Modality Theory addresses your point 2, "the type of information being communicated", i.e. the different forms of expression of information in the media of graphics, acoustics and haptics. When applied to practical interface design in what we call Information Mapping, i.e. the theoretical support of modality choice for specific applications, one has to include consideration of many other factors, among which those you mention, such as pragmatics, user background and manner of use.

Just to briefly relate your claims to the paper below (replacing your `graphics' with `analogue graphics'):

- physical vs. abstract information: this claim is scoped and justified in the paper below with reference to the distinction between analogue and non-analogue representation;

- diverse information vs. related facts: this claim may be too general. For instance, logical relations, being abstract, are better represented through language;

- information on individual objects vs. information on classes of objects: this claim is scoped and justified in the paper below with reference to the distinction between analogue and non-analogue representation. Elisabeth's point about graphics being used to represent classes of objects, and the ensuing ambiguity, is addressed in the paper below through the notion of the `interpretational scope' of analogue graphics.

Lewis Johnson points out the advantages of using combinations of unimodal modalities for presenting information rather than relying on unimodal representations. This claim is strongly supported by the paper which presents the advantages of combining analogue graphics and language into multimodal representations of information. Lewis also suggests that maps constitute a particular type of representation. Modality Theory makes that point. In Modality Theory, tables are not modalities but `modality structures' having particular useful properties. The novice/expert problem wrt. understanding particular representations is addressed in Modality Theory through the notions of `notation' and `arbitrary representations' where users cannot rely on generally known systems of meaning.

The standard presentation of modality theory is: Niels Ole Bernsen: A Toolbox of Output Modalities. It can be found in html at URL: http://www.mip.ou.dk/nis/publications/year97.html

Best

Ole

Niels Ole Bernsen
Why are Analogue Graphics and Natural Language both Needed in HCI?

[Published as: Bernsen, N.O.: Why are Analogue Graphics and Natural Language both Needed in HCI? In F. Paterno (Ed.): Interactive Systems: Design, Specification, and Verification. Focus on Computer Graphics. Springer Verlag 1995, 235-51. Copyright Springer Verlag 1995.]