False.

Multimodal processing of synchronized input is not more demanding.

Multimodal input is more salient and can result in stronger neuronal activity

• super-additivity; The stimulus of a multimodal signal is stronger than the sum of 2 individual modality-specific signals

• reduced variance in object recognition -> increased reliability

-> Human’s perception is designed to deal with multimodal signals

Using multimodal system output will improve … (Jacko et al., 2005)

• Reaction times (perception, not necessarily interaction)

• Salience/prominence (e.g. degree of attention grabbed)

• Decreased workload

• Robustness (e.g. certainty of being perceived at all)

• Naturalness/authenticity?

• Shorter task completion time

E.g. visual system is good for detailed (spatial) information

-> Useful for complex information in HCI

E.g. auditory system is good for general (temporal) information (and those not in view)

-> Useful for warnings, status, attention shifts in HCI

Audio-visual: best for single tasks & normal workload

Tactile-visual: best for multiple tasks & high workload

Key Concepts:

• Limited Resources:

  The model posits that people have a limited capacity for mental processing, like a finite pool of energy. 

• Multiple Resources:

  These resources are not a single, unitary pool, but rather are organized along different dimensions. 

• Resource Dimensions:

  Wickens originally identified three dimensions:

  1. Processing Stage: Different tasks require different stages of processing (e.g., perception, cognition, response). 

  2. Processing Code: Information can be processed in different ways (e.g., verbal, spatial). 

  3. Input Modality: Information can be received through different senses (e.g., visual, auditory). 

  4. Fourth Dimension (Visual Channels):

     Later, a fourth dimension was added to distinguish between focal and ambient vision within the visual modality.

• Red cars are perceived louder as other cars with the same noise -> cross-modal perception, one modality effects the petception of another

• Audio-visual effect on counting (Shams et al. 2000) -> Counting of short events with auditory and visual information mismatch (Audio is dominant) cross-modal perception

• Audio-visual effect on object identification (Sekuler et al., 1997) -> Identical objects pass through each other, but bounce with sound; cross-modal perception

• Audio-visual discrepancy in emotion (de Gelder et al., 1995) -> Emotion in audio effects visual emotion identification (and vice versa) (How often does the bunny jump?); cross-modal perception

Visual effect on sound intensity (Fastl & Zwicker, 2007)

• Red cars are perceived louder as other cars with the same noise

  -> cross-modal perception, one modality effects the petception of another

• Visual Dominance

Audio-visual discrepancy in emotion (de Gelder et al., 1995)

-> Emotion in audio effects visual emotion identification (and vice versa)

-> Examples: Sadness versus Happiness with neutral semantics

-> optimal integration? (brain mixes signals to make a smart decision)

-> cross-modal perception

Audio-visual effect on object identification (Sekuler et al., 1997)

-> Identical objects pass through each other, but bounce with sound

-> Modality Appropriateness

-> cross-modal perception

Audio-visual effect on counting (Shams et al. 2000)

-> Counting of short events with auditory and visual information mismatch (Audio is dominant)

-> For example, if there are several flashes and beeps, people will base their answer more on the beeps.

-> Modality Appropriateness

-> cross-modal perception

Theories:

• Visual dominance

  • In many early studies: vision dominates other modalities (audio, kinesthetic)

  • But: mostly spatial tasks, where vision is exact and reliable

• Modality appropriateness

  • Situational appropriateness determines dominant modality

  • E.g. audio for timing, visual for spatial tasks

• Other integration approaches

  • Optimal integration: a weighting dependent on the estimated accuracy of information

  • might result in anaveraged result

    
    

• Audio-visual location/quality (“McGurk effect”, McGurk und MacDonald, 1976) -> Description: fusion, multimodal integration

  Auditive: /baba/

  Visual: /gaga/ /dada/

  Audio-visual: /dada/ /dada/ (combination, not fusion)

• Audio-visual location (Alais & Burr, 2004) -> the more visual information gets blurred, the more important auditory information gets until a point were both informations are too distant; Multimodal integration -> visual info is not neccesary dominant, we form an optimal integration depending on situation and experience

• Audio-visual location (“ventriloquist illusion”, e.g. Harris, 1965) -> Auditory and visual information are no congruent -> visual information often dominates, auditory information is then adapted (zB. Puppenspieler) Multimodal Intergration

• Visual-kinesthetic (haptic) location (Helmholtz), Multimodal Intergration

  • People were given glasses to distort view and asked to point -> plasticity of the brain to adapt to changes quite rapidly

  • pointing gets distorted, even after taking of glasses -> adaption, kinesthetic (haptic) information of the hands are not just overridden.

  • firstly interpret as “modality dominance” of vision

    
    

• Visual-kinesthetic (haptic) location (Helmholtz),

  • Multimodal Intergration

  • People were given glasses to distort view and asked to point -> plasticity of the brain to adapt to changes quite rapidly

  • pointing gets distorted, even after taking of glasses -> adaption, kinesthetic (haptic) information of the hands are not just overridden.

  • firstly interpret as “modality dominance” of vision -> Visual Dominance

• Audio-visual location (Alais & Burr, 2004)

  
  

  -> the more visual information gets blurred, the more important auditory information gets until a point were both informations are too distant;

  
  

  -> Multimodal integration

  
  

  -> visual info is not neccesary dominant, we form an optimal integration depending on situation and experience

• Audio-visual location (“ventriloquist illusion”, e.g. Harris, 1965)

  -> Auditory and visual information are no congruent

  -> visual information often dominates, auditory information is then adapted (zB. Puppenspieler) Visual Dominance

  -> Multimodal Intergration

• Audio-visual location/quality (“McGurk effect”, McGurk und MacDonald, 1976)

• Description: fusion, multimodal integration

  Auditive: /baba/

  Visual: /gaga/ /dada/

  Audio-visual: /dada/ /dada/ (combination, not fusion)

Multimodal integration refers to the overall process of combining inputs from multiple sensory modalities to create a coherent perception.

Crossmodal perception specifically focuses on how information from one sensory modality can influence the perception of another modality

redundancy worked best -> increased verbal information recall & positive ratings