True or false: Multimodal processing is more demanding than processing with one modality.
False.
Multimodal processing of synchronized input is not more demanding.
Multimodal input is more salient and can result in stronger neuronal activity
super-additivity; The stimulus of a multimodal signal is stronger than the sum of 2 individual modality-specific signals
reduced variance in object recognition -> increased reliability
-> Human’s perception is designed to deal with multimodal signals
What can multimodal system output improve?
Using multimodal system output will improve … (Jacko et al., 2005)
Reaction times (perception, not necessarily interaction)
Salience/prominence (e.g. degree of attention grabbed)
Decreased workload
Robustness (e.g. certainty of being perceived at all)
Naturalness/authenticity?
Shorter task completion time
Each modality is unique (Welch et al. 1986). Which modalities are good for which situation?
E.g. visual system is good for detailed (spatial) information
-> Useful for complex information in HCI
E.g. auditory system is good for general (temporal) information (and those not in view)
-> Useful for warnings, status, attention shifts in HCI
Audio-visual: best for single tasks & normal workload
Tactile-visual: best for multiple tasks & high workload
name all 4 dimensions
Key Concepts:
Limited Resources:
The model posits that people have a limited capacity for mental processing, like a finite pool of energy.
Multiple Resources:
These resources are not a single, unitary pool, but rather are organized along different dimensions.
Resource Dimensions:
Wickens originally identified three dimensions:
Processing Stage: Different tasks require different stages of processing (e.g., perception, cognition, response).
Processing Code: Information can be processed in different ways (e.g., verbal, spatial).
Input Modality: Information can be received through different senses (e.g., visual, auditory).
Fourth Dimension (Visual Channels):
Later, a fourth dimension was added to distinguish between focal and ambient vision within the visual modality.
Give some examples of cross modal perception
Red cars are perceived louder as other cars with the same noise -> cross-modal perception, one modality effects the petception of another
Audio-visual effect on counting (Shams et al. 2000) -> Counting of short events with auditory and visual information mismatch (Audio is dominant) cross-modal perception
Audio-visual effect on object identification (Sekuler et al., 1997) -> Identical objects pass through each other, but bounce with sound; cross-modal perception
Audio-visual discrepancy in emotion (de Gelder et al., 1995) -> Emotion in audio effects visual emotion identification (and vice versa) (How often does the bunny jump?); cross-modal perception
Describe the Visual effect on sound intensity (Fastl & Zwicker, 2007)
Visual effect on sound intensity (Fastl & Zwicker, 2007)
Red cars are perceived louder as other cars with the same noise
-> cross-modal perception, one modality effects the petception of another
Visual Dominance
Audio-visual discrepancy in emotion (de Gelder et al., 1995)
-> Emotion in audio effects visual emotion identification (and vice versa)
-> Examples: Sadness versus Happiness with neutral semantics
-> optimal integration? (brain mixes signals to make a smart decision)
-> cross-modal perception
Describe the Audio-visual effect on object identification (Sekuler et al., 1997)
Audio-visual effect on object identification (Sekuler et al., 1997)
-> Identical objects pass through each other, but bounce with sound
-> Modality Appropriateness
Describe the Audio-visual effect on counting (Shams et al. 2000)
Audio-visual effect on counting (Shams et al. 2000)
-> Counting of short events with auditory and visual information mismatch (Audio is dominant)
-> For example, if there are several flashes and beeps, people will base their answer more on the beeps.
What effects can be seen in studies with discongruent multimodal signals?
Theories:
Visual dominance
In many early studies: vision dominates other modalities (audio, kinesthetic)
But: mostly spatial tasks, where vision is exact and reliable
Modality appropriateness
Situational appropriateness determines dominant modality
E.g. audio for timing, visual for spatial tasks
Other integration approaches
Optimal integration: a weighting dependent on the estimated accuracy of information
might result in anaveraged result
Give Examples for Multimodal Integration
Audio-visual location/quality (“McGurk effect”, McGurk und MacDonald, 1976) -> Description: fusion, multimodal integration
Auditive: /baba/
Visual: /gaga/ /dada/
Audio-visual: /dada/ /dada/ (combination, not fusion)
Audio-visual location (Alais & Burr, 2004) -> the more visual information gets blurred, the more important auditory information gets until a point were both informations are too distant; Multimodal integration -> visual info is not neccesary dominant, we form an optimal integration depending on situation and experience
Audio-visual location (“ventriloquist illusion”, e.g. Harris, 1965) -> Auditory and visual information are no congruent -> visual information often dominates, auditory information is then adapted (zB. Puppenspieler) Multimodal Intergration
Visual-kinesthetic (haptic) location (Helmholtz), Multimodal Intergration
People were given glasses to distort view and asked to point -> plasticity of the brain to adapt to changes quite rapidly
pointing gets distorted, even after taking of glasses -> adaption, kinesthetic (haptic) information of the hands are not just overridden.
firstly interpret as “modality dominance” of vision
Describe the effect of Visual-kinesthetic (haptic) location (Helmholtz)
Visual-kinesthetic (haptic) location (Helmholtz),
Multimodal Intergration
firstly interpret as “modality dominance” of vision -> Visual Dominance
Describe the effect of Audio-visual location
Audio-visual location (Alais & Burr, 2004)
-> the more visual information gets blurred, the more important auditory information gets until a point were both informations are too distant;
-> Multimodal integration
-> visual info is not neccesary dominant, we form an optimal integration depending on situation and experience
Describe the Ventriloquist effect:
Audio-visual location (“ventriloquist illusion”, e.g. Harris, 1965)
-> Auditory and visual information are no congruent
-> visual information often dominates, auditory information is then adapted (zB. Puppenspieler) Visual Dominance
-> Multimodal Intergration
Describe the Audio-visual location/quality effect (“McGurk effect”, McGurk und MacDonald, 1976).
Audio-visual location/quality (“McGurk effect”, McGurk und MacDonald, 1976)
Description: fusion, multimodal integration
What is the difference between multimodal integration and cross modal perception?
Multimodal integration refers to the overall process of combining inputs from multiple sensory modalities to create a coherent perception.
Crossmodal perception specifically focuses on how information from one sensory modality can influence the perception of another modality
Complete this image. Which multimodal instruction works best?
redundancy worked best -> increased verbal information recall & positive ratings
Last changeda day ago