Plan-based Multimedia Interaction

The ability of an intelligent user interface to provide broad and varied information related to the control task must be complemented with an adequate presentation of the information managed by the decision support component. The specific requirements which imply the use of interactive multimedia devices and intelligent multimedia techniques for this purpose are the following:

• adaptability to a wide range of users and presentation situations;
• more effective and appealing presentations through the combination of different communication modes (e.g., text and graphics) so that the strength of one medium will overcome the weakness of another;
• the ability to react by obtaining feedback from the user about the information presented.

The capability to flexibly tailor presentations to a user's individual needs constitutes one of the key aspects of an intelligent multimedia component. Handling dynamic, non-deterministic streams of application data, however, does not only demand for flexible design of media objects rather than canned presentations but also requires dynamic adaption of interaction styles. Multimedia dialogs can be considered as a sequence of communicative acts to achieve certain goals. With the solution proposed in FLUIDS , multimedia interaction is regarded as a non-routine design task and a uniform planning approach is employed for the knowledge-based design of multimedia dialogs. Intelligent multimedia dialog management in FLUIDS comprises the following interrelated design tasks: (1) design of the dialog structure, (2) design of presentation structures for conveying communicative acts, and (3) design of media objects. Knowledge about design techniques is represented by declarative design strategies which are treated as operators of the planning system. Both, design strategies as well as the resulting design structures depend on speech-act theoretic concepts that are adapted from natural language processing to multimedia interaction. Inter- and intra-media coherency relationships are modeled in a generalized version of Rhetorical-Structure Theory (RST).

  

Figure 3: Plan Operator for Describing a Situation

Two examples for presentation strategies are sketched in Fig. 3 and 4. A situation like a line-delay could be presented to the operator by (1) informing him or her about the type of problem using a short spoken message, and (2) describing the involved component. A strategy to inform the operator about a cause is to direct attention to the involved component and to describe its relevant attributes. As these strategies illustrate, presentation planning has to employ temporal reasoning to cope with metric and qualitative temporal constraints for dynamic multimedia coordination.

  

Figure 4: Plan Operator for Describing a Cause

In most cases, the activation of the planning process is query-driven. The dialog manager may receive information from the decision support problem solver in reply to a basic user question or alternativly follow-up questions related to the previous presentation have to be treated. Event-driven behaviour results if the intelligent interface needs to be pro-active to issue alarms, warnings, and messages. Taking this flexibility into account, an important advantage of the plan-based approach to multimedia interaction becomes obvious. Dynamic synthesis of the dialog structure enables adaptive dialog behaviour at run-time.