The previous information may be obtained from different knowledge sources, whose availability depends on the nature of the problem domain, such as the following ones:

The proper exploitation of these sources depends on the way the knowledge they could provide may be used for. For instance, the common sense knowledge has nothing to do with the specification of the supporting domain knowledge but it is essential to define the metalevel knowledge required to select problem solving methods. The right sources to develop a FLUIDS model will be described in the next sections.

Variables and event names together with knowledge bases constitute the conceptual language supporting the conversation in such a way that user and system will communicate, using these language components, to provide mutually concepts and explanations.

This part of the design applies conventional techniques of knowledge acquisition that take advantage of the knowledge level structuring paradigms [Newell, 82]. Some proposals available in literature were already commented in the previous deliverable on FLUIDS architecture (D05.1), where a general format for the Problem Solving Medium was proposed. The main steps that can be identified in the design of the Problem Solving Environment may be the following:

Underlying the design of the Problem Solving Environment is the consideration that a parallelism exists between the problem of selecting problem solvers adequate for a given task from a set of predefined ones and the problem of reusing software modules available in a library. From this point of view, some of the solutions adopted to deal with the reusability problem, regarding the identification of the relevant features to characterize a module or problem solving method and the selection mechanisms that make use of these features, may be incorporated in the design stages of the Problem Solving Medium as it is described below.

2.2.1 Characterization of the components

The key element in the specification of this stage of analysis is the identification, for every type of question, of the main tasks to be performed in order to obtain an answer. This relationship between questions and tasks may be deduced from the parsing of the questions structure.

For instance, to answer a what to do if question, assuming that the answer to a corresponding what is happening type question is already available, the following tasks may be performed:

This example shows how a question requires performing different reasoning tasks to be answered and also, how these tasks may be solved using different approaches.

Then, from this first stage a preliminary description of the Problem Solving Medium is defined as a hierarchical structure of task-methods-subtasks. This structure describes which problem solvers may be in principle capable to perform a task and which subtasks a given problem solver requires other lower level problem solvers realize. This general structure is summarized in figure 3 where the basic domain information supporting the low level subtasks is included at the bottom (in the upper level methods a more abstract description of the domain model is applied).

The figure also shows several problem solvers for the same tasks. Even though in the initial version of any system usually only one problem solver may be provided for every task, the experience in the use of the model and a deeper understanding of the problem domain may lead to introduce additional methods to solve the same kind of problems. For instance, several versions of problem solvers may be designed for efficiency reasons with different conceptual granularity providing in this way different levels of accuracy and detail in the answers. Furthermore, due to the structured top-down organization and the high level specification of the problem solving knowledge, it would be feasible to reuse some of these problem solvers to build new applications.

According to the previous considerations the next steps have to be performed:

The information required to characterize the previous elements may come from different knowledge sources. The tasks and methods considered represent problem solving strategies developed from the experience of domain experts or established from mathematical/physical theories underlying the domain dynamics. At the bottom level, the domain specific information may be extracted from the operators experience or applying data mining processes on data bases that contain system behavior information.

Given that the use of these components is oriented to configure inference structures capable to satisfy users queries, it is necessary to characterize the tasks and the methods with the relevant features that would make them elegible for being part of these structures. In this sense, a conceptual module may be associated to every task where four elements may be distinguished:

Some of the features included in the characterization of a task or method may be refered to the availability of certain information in the working memory of the system. This memory contains the results or assumptions obtained along the problem solving process that are updated in the course of the conversation with the user. The management of this memory is performed by means of a TMS model that is responsible of keeping the consistency of the whole set of assumptions of the system, introducing new ones and deleting those that turn to be false as a result of applying an inference process. The execution of a method generates new information that is justified by the input data and/or some intermediate results that need to be included in the current set of assumptions of the system. The coexistence of this new information with that already present in the working memory requires that no inconsistency can be deduced from this new state of information. The incompatibilities between certain sets of assumptions, also called no-good definitions, may be part of the knowledge model of the TMS or could be introduced by the user during his/her dialogue with the system. Then, the specification of the TMS knowledge model requires the definition of the following concepts:

The no-goods knowledge may be mainly provided by expert operators or obtained from common sense, whereas the classification of the assumptions may be done by the operators together with the knowledge modelers.

2.2.2 Specification of the metalevel knowledge

The specification of the metalevel model implies defining the elements of the previously mentioned conceptual modules associated to the tasks. Two alternatives are proposed:

It should be noticed that this second approach may embed the first one in the sense that the façades of the methods may be part of the specific knowledge of every task, and so the matching rates of the façades with the requirements of the task together with the values of the interaction requirements, the case data and the recent dialogue may be used to make the selection of the appropriate method.

The metalevel knowledge model chosen in the FLUIDS prototypes follows the second approach with the simple classification inference process that once finished, delivers a model of the Problem Solving Medium ready for execution and exploration, and characterized by:

Both components are to be designed adapted to the specific characteristics of a given system. FLUIDS provides the framework where the modeler has to design the domain models and the problem solving methods for these domains and the upper level problem solver together with the frame hierarchy and pattern matching method.

Once the previous knowledge has been defined, it would be necessary the specification of a computational oriented organization of the previous collection of tasks, methods and domain knowledge. The operational version of the knowledge model designed from any of the previous perspectives could be built with the KSM environment, that provides automatic tools to support the basic methods and high level structuring units to configure the knowledge organization. The KSM tool may also provide two different perspectives to configure the required organization:

It is important to notice that even though the case studies in FLUIDS are all based on transport problems, the requirements for the target conversation model previously described show that the FLUIDS framework is designed abstract enough to deal with any system which evolves along time and where knowledge is available to define a platform of advanced concepts for decision making derived from the basic variable values provided by the underlying data system.

2.2.3 Summary

According to the previous description, the different activities required to design a Problem Solving Medium may be summarized as follows:

Figure 6 presents a summary of these steps.

The mere display of derived information is not sufficient to provide a user with an adequate presentation of the results obtained by the Problem Solving Medium in reply to a specific user query. Rather, a generative multimedia-based presentation system is required whose behavior needs to be described by more than just a simple one-to-one mapping of input units to media objects which will be displayed to the user. To ensure that the generated multimedia presentations are understandable and effective, an automated presentation system needs to be intelligent in the sense that it is able to make appropriate design decisions based on presentation knowledge and contextual knowledge, and to manage the various interdependencies between choices. For an intelligent multimedia presentation system as defined in [Bordegoni, 97] the attribute effective means that the particular information needs of the individual users are best met under a given set of presentation constraints, such as resource limitations and the user's knowledge and style preferences.

The presentation task within a FLUIDS-based intelligent user interface is essentially a complex information transformation process which translates application data and knowledge into presentations composed of media objects which are to be consumed by the human user of the system. Consequently, the design of the presentation model for a specific FLUIDS application aims at two major goals:

The subsequent sections present in more detail a suitable methodological approach for carrying out this design process.

2.3.1 Elements of the Design Process

An intelligent multimedia presentation system is a knowledge-based system that employs multiple media to present information to the user. The primary input for such a generative presentation system form presentation goals which have to be achieved by the system, possibly accompanied by a set of presentation constraints affecting the presentation process. Application data and knowledge necessary for achieving posted goals is part of the input to the presentation system and provides the semantic grounding of each presentation which may be generated by the system. Following this view, there is a clear division of labour between the presentation subsystem and the other components of a FLUIDS interface in terms of the classical "What?" vs. "How?" distinction. Whereas the Conversation Model and Problem Solving Medium are concerned with the determination of content ("What to communicate?") the Presentation Model has its focus on the form of conveying information ("How to communicate?").

Regarding the architecture of the multimedia presentation component it is appropriate to differentiate between presentation generation, i.e. the automated synthesis of a multimedia presentation, and a separate layer of presentation display. The output of the core generator is an unrendered presentation, a so-called presentation script, encoded in a declarative representation format. The presentation display component (or presentation runtime environment) converts this intermediary representation into a presentation perceivable by the user. This means that in order to execute it, each part of the unrendered presentation is dispatched to the interface of a suitable media-specific display module.

The generation task is guided by the idea that information presentation can be characterised as a goal-directed activity aiming at the realization of a complex communicative act. Hence, a uniform planning approach is employed for the knowledge-based design of multimedia presentations. Knowledge about design techniques is represented by declarative design strategies which are treated as operators of the planning system. Both, design strategies as well as the resulting design structures depend on speech-act theoretic concepts that are adapted from natural language processing to multimedia interaction. The result of the top-down planning process is a design plan with a hierarchical structure which is generated through the recursive decomposition of the complex communicative act corresponding to an intentional goal to be achieved. A particular advantage of the plan-based approach lies in its inherent support for better adaptivity. Taking into account user characteristics and preferences as explicit presentation constraints allows to flexibly tailor presentations to a user's individual needs.

In addition to generating a multimedia presentation for the information delivered by the Conversation Model and the Problem Solving Medium the presentation planner is also responsible for the dynamic design of the dialogue structure. Declarative dialogue strategies are used to determine which follow-up questions and new potential queries need to be activated together with the generated presentation.

According to the various aspects mentioned before the design of the Presentation Model comprises two main phases:

The conception phase includes the detailed description of what the final presentations should look like, the principal organisation of the user-system dialogue, the identification and classification of relevant user attributes and their effect on the output, and finally the specification of the corresponding presentation scripts. The operationalisation covers engineering of the design knowledge (presentation strategies and dialogue strategies) and required extensions of presentation display modules. The different tasks that need to be accomplished during the design of the Presentation Model lead to a further modularisation of the basic design phases. From this perspective, each task corresponds to a separate step within the overall design process proposed here.

2.3.2 Conception of the Intended Presentation Results

The conception phase aims at a detailed description of the required input-output behaviour of the information presentation component. It can be split up into four successive stages:

Starting from the specification of the Conversation Model, which defines the possible questions to and answers from the decision support system, the first step is to select suitable output modes and media combinations for the specific application and to sketch the presentations that should be provided to the user for each type of potential query result. This includes also possible variations of presentation output. Initial design specifications concerning the different multimedia presentation formats rely on paper prototyping. Although the final version of the Conversation Model is not required, in order to start this activity it is assumed that an in-depth design for the most important task scenarios already exists. Such a task scenario is a complete interaction sequence from problem recognition to problem treatment. It is advisable to provide real sample data which may be used as test cases for the different task scenarios.

The use of scenario building and storyboarding during the treatment of the selected task scenarios also provides helpful hints for the second step, the determination of the desired dialogue behaviour. The dialogue structure specifies for every system response what kind of follow-up questions and new potential queries will be enabled within the user interface. Careful organisation of the dialogue is required to effectively guide the operator through the conversation without hindering his or her efficiency in performing the control task. Usually, some initial requirements concerning dialogue formation are already captured during the design of the Conversation Model. A good way of illustrating the results of this design activity are dialogue graphs.

A third stage of the conception phase is devoted to user adaptability. The identification and classification of user characteristics is needed to make those features explicit that control presentation generation in a user dependent way. Explicit user stereotypes and profiles enable different presentation formats for the same decision support information depending on the kind of user and his or her preferences. The user stereotypes employed in a concrete FLUIDS system correspond to a coarse classification of potential end users into different categories (like for example 'occasional user' or 'expert'). Since such a distinction mainly relates to the question of "What information has to be provided?'' as opposed to "How should the information be presented?'' the classification will be made explicit during the design of the Conversation Model. From the point of view of the multimedia presentation component the different user categories have an important influence on the design of the dialogue structure (e.g., for a novice user some additional types of follow-up queries would be enabled as well). Furthermore, a stereotype provides sensible default values for initializing user profiles which capture individual preferences. Such a user profile constitutes a kind of a feature vector which contains specific settings for all relevant dimensions. Regarding the multimedia presentation component, the user profile includes feature values that control variations of the presentation behaviour. Important aspects include constraints and preferences concerning the use of specific media and media combinations (e.g. 'no speech output', 'few animations', 'detailed textual information'). Feature variables can later be used as parameters within the plan operators of the presentation planner in order to guide the selection of the most appropriate presentation strategy. These feature variables and their potential values can be identified from the variations of the sketched presentations and the dialogue behaviour that have been derived during the first two stages.

The complete information transformation process is divided into the automated generation and the actual display of a multimedia presentation. The last step of the conception phase of presentation model design is concerned with the exact specification of the intermediary result of this transformation process. The output of the presentation generator is a presentation script in textual notation that can be executed by the presentation display component. Such a script contains specifications of displayable media objects (like a piece of written or spoken text, a specific graphical display or animation element, etc.) together with layout information. It entails all the information - including in particular temporal parameters - required to run the presentation properly. The notation used for presentation scripts constitutes a sort of specification language which can be easily extended with new types of media objects. The need for such extensions of the basic vocabulary is typically driven by application-specific graphical elements (e.g. 'show-network') that provide a higher level of abstraction concerning the graphics operations to be employed for presentation planning. These additional primitive media objects encapsulate domain specific data access and drawing functions. The identification of required extensions and hence the careful determination of the appropriate level of granularity for these primitives is one of the important issues for the final design step of the conception phase. For every presentation result and all its variations, as they have been sketched during the first stage of the conception phase, a sample presentation script needs to be written in order to clearly specify the intended output of the generation process. The presentation display component of the FLUIDS software environment can be used in stand-alone mode for the interactive development and testing of the multimedia presentation examples. In order to fully exploit such a RAD approach (Rapid Application Development) it can be useful to start rather early with the realisation of display extensions as described in the next section.

Once the conception of the desired information presentation behaviour is available, the second phase of the design process - the operationalisation - can be started to complete the design of the presentation model of the new application.

2.3.3 Operationalisation of the Presentation Process

The goal of the second design phase is to specify the presentation process in operational form. Taking into account the modularisation of the multimedia presentation process three different tasks can be distinguished that may be carried out concurrently:

The presentation generator, a parallel top-down planning module, applies the design knowledge encoded as declarative presentation strategies to build up a coherent and temporally coordinated presentation for a specified communicative goal. In general, this communicative goal is to inform the user about the result obtained from the Problem Solving Medium in reply to his or her last query.

The planner tries to find an applicable presentation strategy, i.e. plan operator, for the given communicative goal and incrementally generates a refinement-style plan in the form of a directed acyclic graph. This hierarchically structured plan for the presentation to be generated reflects the propositional contents of the potential presentation parts, the intentional goals behind the parts, and rhetorical relationships between them. Following this kind of speech-act theoretic approach, inter- and intra-media coherency relationships are modeled in a generalized version of Rethorical-Structure Theory (RST). Examples of such semantic and pragmatic coherence relations are 'Sequence', 'Motivation', 'Elaboration', 'Enablement', and 'Summary'. It should be noted that the output of the generator cannot be said to be the complete internal representation of a presentation. Only the leafs of the plan structure correspond to elements within the resulting presentation script.

Presentation strategies represent knowledge concerning how to decompose a given presentation task into subtasks or, in case of elementary subtasks, which media objects should be used to convey the subtasks. A presentation strategy consists of:

Although generic presentation strategies can simply be reused, a basic repertoire of presentation strategies needs to be defined in order to capture application-specific requirements. The design of these strategies is a knowledge engineering activity quite similar to what needs to be done for the Conversation Model and the Problem Solving Medium. The design results regarding the Conversation Model (and Problem Solving Medium) specify the knowledge structures that define the top-level input from which the presentation strategies have to start. The results gained from the conception phase determine the desired output of applying the presentation strategies and define feature variables that can be utilized to constrain strategies according to user stereotypes and specific preferences. For each given type of presentation it is a good idea to first note down the correlations between the different elements of the input knowledge structure and the resulting presentation script. This provides an initial sketch of the overall presentation plan which can be further elaborated to identify the necessary decomposition rules.

The second task to be performed for the engineering of the presentation design knowledge is the specification of dialogue strategies on the basis of the dialogue structures that have been determined during the conception phase. Although the same planning mechanism is used, dialogue strategies are usually much simpler than normal presentation strategies since temporal and layout constraints as well as rhetorical relationships do not have to be considered. A dialog strategy defines a rule for deciding which follow-up questions and new queries have to be activated depending on the last query, the query result, and the characteristics of the current user. The output of dialogue planning are declarative specifications of so-called command objects. Such a command object is a more abstract description of potential user queries which is independent of a specific input modality (like speech, pointing, or menue item). The runtime system of the multimedia user interface is responsible for mapping user input to the appropriate command object. Quite often it is possible to factor out common dialogue structures so that dialogue strategies can be organised in a hierarchical manner.

A last still missing stage of the operationalisation phase is the design of the required presentation display extensions. Within the FLUIDS software environment a specification of a displayable media object as part of a presentation script is mapped onto an instance of a corresponding Java object that can be executed by the Java-based presentation display engine. New media object types are easy to define by extending and reusing the FLUIDS Java Class Libraries which provide the underlying multimedia presentation display framework.

2.3.4 Summary

The design of the Presentation Model is an iterative process where each cycle is composed of (1) a conception phase, and (2) an operationalisation phase. Several stages can be distinguished within these phases and the following elements characterise the target of each step:

Initial design specifications concerning the different multimedia presentation formats rely on paper prototyping. In this context, tools and techniques for user analysis and activity analysis like interviews and group discussions can be helpful for the determination of user categories. In order to specify the desired dialogue behaviour for the user interface scenario building and storyboarding are employed. Later on, RAD techniques (rapid application development) can be exploited for refining these draft design specifications by trying out alternative presentation formats. FLUIDS' Java-based presentation display engine provides a scripting facility which enables rapid application development through the specification of hand-crafted presentation scripts. These early mock-ups provide valuable specifications for the formulation of presentation strategies. Finally, walkthrough and co-operative evaluation techniques as well as user trials and field trials are applied to further elaborate the design specification of the multimedia presentation component.

A guiding principle for this generic process is the iteration of design solutions. Taking into account the interdependencies between the various design tasks the different steps of the design process can also be carried out in a different order. For example, the specification of presentation scripts (1.4) depends on the first task, i.e. a sketch of presentation results, but not on the determination of dialogue structure and user characteristics. In principle, all tasks of the operationalisation phase are independent from each other and rely only on corresponding stages of the conception phase. Once again, the proposed design methodology is independent of a specific application domain.

This demonstrates that the FLUIDS conception is a framework for developing applications by fulfilling the parameters of the underlying generic model which in this case is a complex task because explicit design knowledge needs to be formulated. This gives generality at the cost of detailed formulation of the declarative parts of the Presentation Model.

Figure 7 summarizes the necessary components to specify a FLUIDS model according to the design considerations described above together with the knowledge sources that may be used to define these components.

To design a FLUIDS model users and experts knowledgeable of the application objectives must create a model with the three components described providing the adequate knowledge for the different models. The final operating version is obtained through the following iterative process (see figure 8):

As every model test provides with an explanation associated to the set of answers generated, the expert users may criticise the results and propose refinements of the different problem solving and domain models.

This process of refinement may be started first by independent models and, after, by integrating sequentially the operation until a satisfactory global performance is obtained. This allows to work in parallel in different models until the integration phase.

The generality of the FLUIDS approach can be demonstrated analyzing one by one the nature of the collection of concepts summarized in figure 7:

Even though the flexibility of the FLUIDS approach is demonstrated by means of the three prototypes, that show how a high level decision support model can be applied to develop solutions to different traffic management problems; next this demonstration is reinforced with the presentation of an example of application to the domain of emergency management.

In this case, the goal is designing a decision support system to help operators to take decisions in the case of floods emergencies. For emergency management, a valid reasoning strategy is to identify the undesirable situations, to analyze their causes and to act on these causes and/or on the predictable effects in such a way that, modifying the causes, the undesirable impacts may dissapear or at least may be alleviated. This reasoning sequence allows to provide answers to the three classes of questions considered in FLUIDS: what is happening, what may happen and what to do. In particular:

In order to answer these questions, three main knowledge bodies may be required:

A proposal of the Problem Solving Medium required to support the previous model is presented in figure 9. As can be observed in this figure, the methods considered to define the model are independent of the type of emergencies that may need to be managed.