Plan-based Multimedia Interaction

With the solution proposed in FLUIDS , multimedia interaction is regarded as a non-routine design task and a uniform planning approach is employed for the knowledge-based design of multimedia dialogs. More specifically, intelligent multimedia dialog management comprises the following interrelated design tasks:

1. design of the dialog structure,
2. design of presentation structures for conveying communicative acts, and
3. design of media objects.

Given this, the PrePlan planning component plays a central role within the dialog manager. Each processing cycle leads to performance of the different design steps which are organized into a task pipeline.

Knowledge about design techniques is represented by declarative design strategies which are treated as operators of the planning system. Both, design strategies as well as the resulting design structures (see Fig. 11) depend on speech-act theoretic concepts that are adapted from natural language processing to multimedia interaction.

 

Figure 11: Design Plan for a Multimedia Presentation 

 

Figure 12: Plan Operators for Describing a Delay Situation 

   (define-plan-operator
     :HEADER (A0 (PLAN-PRESENTATION P A  What-Is-Happening ?query)) 
     :CONSTRAINTS (BEL P (?query line-status non-uniform-delay))
     :INFERIORS ((A1 (CREATE-BACKGROUNDS-GRAPHICS P A ?query))
                 (A2 (INFORM-ABOUT-DELAY P A ?query)))
     :QUALITATIVE ((A1 (m) A2))
     :START A1
     :FINISH A2)
 
   (define-plan-operator
     :HEADER (A0 (CREATE-BACKGROUNDS-GRAPHICS P A  ?query)) 
     :CONSTRAINTS (*AND* (BEL P (?query text ?text-window))                     
                         (BEL P (?query graphic ?graphic-window)))
     :INFERIORS ((A1 (S-CREATE-WINDOW P A ?text-window ?graphic-window))
                 (A2 (S-SHOW-LINE P A ?graphic-window ?line))
                 (A3 (S-SHOW-LINE-STOPS P A ?graphic-window ?line)))
     :QUALITATIVE ((A1 (m) A2) (A2 (e) A3))
     :START A1 :FINISH A3)

   (define-plan-operator
     :HEADER (A0 (INFORM-DELAY-DETAILED P A ?text-window
                                            ?graphic-window ?y-pos-1 ?y-pos-2
                                            ?vehicle ?v-location ?v-delay 
                                            ?delay-label))
     :INFERIORS ((A1 (S-SHOW-VEHICLE P A ?graphic-window ?vehicle
                                         ?v-location ?y-pos-1))
                 (A2 (VERBALIZE-VEHICLE P A ?text-window ?vehicle))
                 (A3 (S-SHOW-RED-BLINKER P A ?graphic-window ?v-location 
                                             ?y-pos-1))
                 (A4 (VERBALIZE-VEHICLE-DELAY P A ?text-window ?minutes))
                 (A5 (S-SHOW-LABEL P A ?graphic-window ?delay-label 
                                       ?v-location ?y-pos-2)))
     :QUALITATIVE ((A1 (e) A2) (A2 (s) A3) (A2 (m) A4) (A4 (m) A5))           
     :METRIC ((20  <= DURATION A3 <= 30)) :START A3 :FINISH A3)

Some examples for presentation strategies that are specific to the public transport application are shown in Fig. 12. These operators can be applied for the presentation of a problem situation in which some vehicles on a specific line are delayed. As these strategies illustrate, presentation planning has to employ temporal reasoning to cope with metric and qualitative temporal constraints for dynamic multimedia coordination.