False.

Multimodal processing of synchronized input is not more demanding.

Multimodal input is more salient and can result in stronger neuronal activity

• super-additivity; The stimulus of a multimodal signal is stronger than the sum of 2 individual modality-specific signals

• reduced variance in object recognition -> increased reliability

-> Human’s perception is designed to deal with multimodal signals

Using multimodal system output will improve … (Jacko et al., 2005)

• Reaction times (perception, not necessarily interaction)

• Salience/prominence (e.g. degree of attention grabbed)

• Decreased workload

• Robustness (e.g. certainty of being perceived at all)

• Naturalness/authenticity?

• Shorter task completion time

E.g. visual system is good for detailed (spatial) information

-> Useful for complex information in HCI

E.g. auditory system is good for general (temporal) information (and those not in view)

-> Useful for warnings, status, attention shifts in HCI

• Red cars are perceived louder as other cars with the same noise -> cross-modal perception, one modality effects the petception of another

• Audio-visual effect on counting (Shams et al. 2000) -> Counting of short events with auditory and visual information mismatch (Audio is dominant) cross-modal perception

• Audio-visual effect on object identification (Sekuler et al., 1997) -> Identical objects pass through each other, but bounce with sound; cross-modal perception

• Audio-visual discrepancy in emotion (de Gelder et al., 1995) -> Emotion in audio effects visual emotion identification (and vice versa) (How often does the bunny jump?); cross-modal perception

Theories:

• Visual dominance:

  • In many early studies: vision dominates other modalities (audio, kinesthetic)

  • But: mostly spatial tasks, where vision is exact and reliable

• Modality appropriateness

  • Situational appropriateness determines dominant modality

  • E.g. audio for timing, visual for spatial tasks

• Other integration approaches

  • Optimal integration: a weighting dependent on the estimated accuracy of information

  • might result in anaveraged result

• Audio-visual location/quality (“McGurk effect”, McGurk und MacDonald, 1976) -> Description: fusion, multimodal integration

  Auditive: /baba/

  Visual: /gaga/ /dada/

  Audio-visual: /dada/ /dada/ (combination, not fusion)

• Audio-visual location (Alais & Burr, 2004) -> the more visual information gets blurred, the more important auditory information gets until a point were both informations are too distant; Multimodal integration -> visual info is not neccesary dominant, we form an optimal integration depending on situation and experience

• Audio-visual location (“ventriloquist illusion”, e.g. Harris, 1965) -> Auditory and visual information are no congruent è visual information often dominates, auditory information is then adapted (zB. Puppenspieler) Multimodal Intergration

• Visual-kinesthetic (haptic) location (Helmholtz), Multimodal Intergration

  • People were given glasses to distort view and asked to point -> plasticity of the brain to adapt to changes quite rapidly

  • pointing gets distorted, even after taking of glasses -> adaption, kinesthetic (haptic) information of the hands are not just overridden.

  • firstly interpret as “modality dominance” of vision

    
    

cross-modal perception: ask for one modality

multimodal integration: ask multimodal

Multimodal integration refers to the overall process of combining inputs from multiple sensory modalities to create a coherent perception. Crossmodal perception, on the other hand, specifically focuses on how information from one sensory modality can influence the perception of another modality