• Visual (Eyes)

• Auditory (Ears)

• Haptic (Skin)

• Olfactory (Nose)

  • Gustatory (Tongue)

• Vestibular (Equilibrium org., Balance)

Difference: Multimodal systems transform information automatically and process abstract information

-> A multimodal system seems to be more “intelligent”

• “active-input mode”: dialog systems

• “passive-input mode”: adaptive and context aware systems

• virtual environment

CARE properties (Coutaz et al., 1995):

• Complementarity: Different modalities have to be used in order to reach the goal.

• Assignment: Only one modality is selected, either by the system or the user.

• Redundancy: Different modalities are used, bearing comparable information, either in parallel or sequentially.

• Equivalence: Any available modality can be used. There are no restrictions on the temporal order.

Modality relations

CASE properties / multimodal design space (Nigay & Coutaz, 1993):

• Exclusive: No fusion possible, only one modality at a time.

• Alternate: Modalities are fused sequentially.

• Concurrent: Different modalities are used in parallel, but are interpreted independently.

• Synergistic: Modalities are used in parallel with fusion.

• Levels of abstraction: signal level or semantic level

• Provide appropriate input and output modalities for each piece of information

• Offer flexibility for different user groups, environments etc.

• -> balancing user deficits, contextual limitations

User input for multimodal systems (not multimedia systems!) can be recognized better than with simpler systems due to

• synergies in the fusion module (-> better speech recognition with face and gesture recognition improves performance)

• user choice of expected most robust modality

• user habit of speaking more simple in multimodal interactions

In contrast to Multimedia Systems, users can interact naturally

• “Use” affect, emotions

• Use accompanying “natural” hand gestures

• “natural” body language (e.g. body direction) is processed

Users seem to prefer multimodal interaction.

• especially, when the cognitive load increases (time pressure, task difficulty)

• one reason might be the usage of different cognitive resources for separate modalities è Processing of information is faster and less demanding (Wickens, 1999

• Embodied Systems (animated talking heads, avatars) are assumed to enable natural and intuitive interaction

• can positively affect user judgement and interaction

Almost natural virtual humans are rated much worse than not so natural ones

Performance in the presence of other users:

- Task performance increases for easy tasks

- But decreases for complex tasks

Most likely due to increased attention, e.g. faster decisions

• Equal level of intimacy

• Balanced on modalities

• E.g.: closer distance, less gaze