2.1 Terms related to Media and Modality

It cannot be denied that there is a lot of confusion about the fundamental notion of media. One reason for this misunderstanding is certainly due to the fact that the term media is used with different meanings in different contexts, such as in semiotics, psychology, telecommunications, or computer science. The closely related term modality is a further source of confusion. Some authors seem to use both terms as synonyms, while others tend to reserve modality only for input (to be processed by a machine interpreter), and vice versa, media only for output (to be produced by a machine generator). Each view has its pros and cons, depending on the aspect under consideration. It is not our intention to provide the ultimate solution to the medium/modality-debate across disciplines. Rather, our use of terms resulted from a pragmatic ``merge'' of different approaches. Our view of the medium/modality distinction is illustrated in Fig. 2.

 

Figure 2: The Media/Modality Distinction 

The leftmost column represents a human user sensing the environment which, among other things, may include output delivered by dedicated display devices. Taking this view, we associate different meanings with the notion medium depending on the particular focus or perspective:

medium as physical space to realize perceptible entities

With the human sensory apparatus as ``target system'', we can consider a medium as being a certain physical space in which perceptible entities are realized. As there are different types of perceptibles, say visibles, audibles, hapticables, olfactables and so forth, one may use these terms for differentiating between media as well. However, further refinements can be made, e.g., to differentiate between mono- and stereo visibles or audibles. In any case one may characterize a medium among the physical dimensions which are relevant to realize a particular perceptible entity.

medium as a type of information and/or representation format

In this view, medium refers to a certain type of information and/or the representation format in which information is stored. Examples are pixmap graphics, 3D-scene descriptions, sequences of video frames, audio data, etc.. It is assumed, that for each medium there is a dedicated physical device which is able to produce perceptible entities. In other words, devices serve to display media objects (before the user's eyes, ears, etc.). Apart from screens, projectors, printers, loud-speakers, etc., there are also more futuristic examples of devices such as spatial light modulators for electronic holography, or devices which can produce smells. A framework for a feature-based characterization of media types is currently under development in the context of the PREMO standardization activity.

Less surprisingly, the term multimedia refers to the use of multiple media. However, one has to take into account which meaning of media is meant. In the first case, one may think of a common physical space in which different perceptible entities can be realized. Examples include the simultaneous production of visibles and audibles. In the second case, multimedia stands for a composition of some basic media types. For example, one may compose a medium video-with-sound of the two media video and audio. A characterization of composed media may include additional properties such as temporal relationships between the involved media.

A further important distinction is that between types and instances. Again, what we understand as medium is decisive. When considering medium as a physical space, instantiations are particular perceptible entities. For example, a beep is an instantiation of an audible. In the other reading of media, an instance corresponds to a particular set of data in a certain format, for example the ascii source file of this document is a media object.

During the last years, research in multimedia systems has concentrated on the representation, creation, storing, conversion, distribution, and display of media objects. When dealing with IMMPSs, however, the information content of a media object and the way it is encoded have to be addressed, too. The right most column of Fig. 2 illustrates what is meant by information content. Assuming a human author, information content may be an intellectual entity existing only in the author's mind. Assuming authoring will be done by a computer system, there must be an explicit representation of the information content. As these representations are usually not in a format presentable to human users, we may speak of raw data, or of knowledge structures in the case of highly structured representation formats. In order to bridge between information content on the one hand, and media objects on the other, we introduce the following terms:

modality

refers to a particular way or mechanism of encoding information which is to be presented to humans or machines in a physically realised intersubjective form. Examples include 2D- and 3D graphics, written and spoken natural language, music and earcons. For a more fine-grained classification of presentation modalities we refer to a paper by Bernsen (see this issue).

multimodality

refers to the use of multiple modalities to encode information. For example, this proposal is a bimodal presentation since it utilizes written text and 2D diagrams to describe the reference model. Note that a single medium may be used as a common physical realization of several modalities, e.g., the medium audio for spoken natural language accompanied with some background music.

In the context of this paper the ultimate goal of using media and modalities is to communicate information to the user. However, the use of media/modalities may be further distinguished with regard to different aspects, e.g., sequential vs. parallel use with regard to time, or redundant vs. complementary use with regard to the encoding of information, or informative vs. decorative use with respect to the communicative function.

Concerning the relationship between media and modalities, it has to be noted that the choice of a certain modality also places a restriction on the medium to be used. Moreover, in many concrete system implementations, there is even a one-to-one mapping from a certain modality to a certain type of media objects. In such a case, it does not bother so much if modality and medium are mixed up. In the following, we introduce a few more terms related to media and modalities:

presentation

will be used as a general term for any composition of media objects which have been created for the purpose of communicating information to a user. Depending on context, we will use the term ``presentation'' either to emphasize the material aspect of media objects (i.e, presentation = document), or to emphasize the activity of communicating information to a user.

multimedia/modal referring expression

a composition of at least two different media/modes objects included in a presentation with the intention to refer to an object or a subject matter in the application domain.

cross-media/modal reference

an expression included in a presentation with the intention to refer to another part of a presentation (e.g., text in the caption of a graphic that refers to a subpart in the graphic). It is assumed that the expression and the referenced part are realized in different media/modalities.