Kapitel 4 Acoustic and haptic modalites

acoustic language using analogue units >> analogue units sound somewhat similar to what they represent (e.g. onomatopoeia >> click, bang)

acoustic language using non-analogue units >>

• non-analogue units are symbolic or arbitrary

• basically all spoken language with small contributions

• only understood when single individual is speaking

• situated communication or linguistic communication

• natural to the hearing since learning it from an early age

analogue static-dynamic acoustics >>

• help to express dimensionality/ spatial information (e.g. hearing if a plane is approaching or leaving based on different sound source distributions)

• help to auralise information about objects (real or imagined), processes, events

• largely dynamic representation >> static only as a limiting case (e.g. someone turning off a repetitive sound)

haptic language using analogue units >> expresses information by means of analogue haptic elements used as units of linguistic expression

haptic language using non-analogue units >> is haptic language in general (except for small contribution with analogue elements)

analogue static-dynamic haptics >> representing the detailed tactile feel of real and imagined objects, their physical state, texture, structural detail, changes and patterns of movement

acoustic language using analogue units -> Onomatopoeia (“Lautmalerei”), e.g. “wauwau” instead of dog

• Language expressed through speech and meant to be perceived primarily through hearing

• Developed for situated communication among, and with, the hearing

• Natural to the hearing (it has been learnt from a very early age)

• Exists in many different natural language tongues

• In native and known languages, have very high saliency to humans -> distingush it in a crowed room with people speaking different languages

• Information channels (in addition to acoustics): voice quality, stress, intonation, dialect, accent, personality

• Situated communication with the hearing who have skills in generating or interpreting a spoken language, or who wish to learn that language

• Linguistic communication with the very large majority of hearing people (including the illiterate)

  • when hands and eyes are occupied

  • when graphic or haptic language materials are unavailable / inaccessible (no light, gloves on) / impractical (outdoor activities)

• Attracting human attention even in relatively noisy environments

• Only understood when a single individual is speaking at a time

• is a perceptual phenomenon that demonstrates an interaction between hearing and vision in speech perception

• The illusion occurs when the auditory component of one sound is paired with the visual component of another sound

• The visual information a person gets from seeing a person speak changes the way they hear the sound

• If a person is getting poor-quality auditory information but good-quality visual information, they may be more likely to experience the McGurk effect

Example: da-da-da / wa-wa-wa / ba-ba-ba

• with analogue static-dynamic acustics

Example: Dopplereffect

• Tennis -> you can feel if you hit the ball in the right way

• Object you can’t see -> you still feel its shape, weight, rigid or soft, how it reacts to tilting, swinging or squeezing

• Situated haptic output language like speech and sign language -> relatively rare form of linguistic expression is primarily aimed at someone who is blind and deaf-mute at the same time

• Non-situated haptic output language for non-situated communication through tactile perception, primarily among, and with, the blind and hard-of-seeing (e.g. Braille)

-> Equivalent to alphabetic graphic written language -> haptic output language (non-situated), using non-analogue units of expression, developed for communication among and with the blind and hard-of-seeing

• haptic language using non-analogue units

Choosing Modalities (Requirements):

1. be able to express the information to be exchanged between user and system -> we need a modality (some make more sense than others)

2. express the information as aptly as possible (always different modalities to choose from -> some are better suited)

   1. unless other factors count against it, choose apt candidate modalities (e.g. User (disablilities / compencies / knowledge), Task (privacy / surroundings / distraction / loud), Tools (unavailable, broken))

   2. if a modality is conditionally apt, try to complement it with another (e.g. sound when computer finished booting -> bc just displaying is not very usefull when not being infront of pc)

   3. if an apt modality cannot used look for equally apt alternatives

Excluding Modalities:

• User disabilities Haptic text input with keyboard no usable for paralyzed people.

• Lack of sources and resources Photo of the incident is not available.

• Technical limitations and unavailability No high-resolution haptic display available.

• Not enough output space Limited screen space requires icons instead of text.

• Challenge the users Making games harder by using sounds instead of keywords (e.g. show where enemies are in a shoorter game) (info is hard to discover -> can it be used to cheat? -> is it cheating if it’s part of the game)

-> spoken language; or speech modality in general

• language expressed through speech and meant to be perceived primarily through hearing

• Developed for situated communication (hearing)

• Natural to the hearing (learned from young age / birth)

• Exists in many different natural language tongues

• In native and known languages, have very high saliency to humans

• Information channels (in addition to acoustics): voice quality, stress, intonation, dialect, accent, personality

• Situated communication with the hearing who have skills in generating or interpreting a spoken language, or wish to acquire such skills

• Linguistic communication with the very large majority of hearing people (including illiterate)

  • when hands and eyes are occupied

  • when graphic or haptic language materials are unavailable / inaccessible (no light, gloves on) / impractical (outdoor activities)

• Attracting human attention

• Only understood when a single individual is speaking at a time

Example: BaBaBa - Video, cocktail party effect

• express specific information through analogue representation in the acoustic medium

  • largely consist of dynamic representation although having static acoustic representation as a limiting case, such as a repetitive sound that can only be turned off by the user

  • can to some extent represent 2D and 3D spatial information

• Analogue acoustics are apt for auralising information about real or imagined objects, processes, and events or aspects of these

Example: Dopplereffect airplane / ambulance

• quite hard to find sth that is using analogue units in haptic language -> mostly linked to our language (non-analogue; e.g. Braille converts alphabet into non-analogue haptic units)

• express information by means of analogue haptic elements used as units of linguistic expression.

• might play a role in haptic languages for the blind-deaf-mute, for instance by facilitating fast initial language learning

Haptic Language consists of three very different types:

• Haptic input language All forms of non-situated language produced haptically (e.g. hand-writing, typing, morse key tapping)

• Situated haptic output language like speech and sign language -> relatively rare form of linguistic expression is primarily aimed at someone who is blind and deaf-mute at the same time

• Non-situated haptic output language for non-situated communication through tactile perception, primarily among, and with, the blind and hard-of-seeing (e.g. Braille)

• Haptic output language (non-situated) can be assumed to be apt for non-situated communication for the blind and hard-of-seeing (if established, like Braille)

• Haptic language is only apt for recipients (people and machines) who understand the particular natural language or notation used and have skills in decoding haptic expressions

Note: percentage of blind and hard-of-seeing who are familiar with Braille varies widely (>10% to < 80%)

Example: Braille, protectile ASL

Analogue haptics are

• are haptic representations in 1D, 2D, 3D (and time) & have some similarity with what they represent

• express information but limitations are dimensionality, specificity, sketchiness, aspects of similarity, and resolution of the representation

• include static and dynamic haptic representation

• mostly consist of static information representation for the time being -> i.e., representations that allow unrestricted freedom of tactile inspection of the analogue information represented

• representing the detailed tactile feel of real and imagined objects, their physical state, texture, structural detail, changes and patterns of movement

• representing overviews of objects as well as event and process snapshots in 1D, 2D and 3D spatial

Drawbacks:

• overviews must be built step by step through exploration; cannot be acquired at a glance like when built through vision

• unwieldly to inspect large and complex objects

• Some information channels are missing, e.g. color

• Dynamic haptic objects, events and processes are difficult to inspect

Example: phantom device (3D mouse)

• complementary: figure + subtitle -> describing what you see; professor presenting slides

• addition: voice chat in video game

• redundancy: apple pay -> vibration + show on display; stop sign

• elaboration: screen in bus + automatic voice -> next stops + description on screen; buying shoes with voice assistant (+display); paper + talk

• alternative: movie + transcript; iDrive: adjust volume (knob, gesture, steering wheel knob)

• stand-in: internet is broken -> no insta :( + text that sth is wrong; fail to explain color to a blind person / closed caption describing actions in movies

• substitution: audio discription of movie scenes

• conflict: every pattern can lead to a conflict if it’s failing -> e.g. wrong bus stop and othe bus stop is read out loud

• many, if not all, spoken natural languages include a subset of words or other elements that have some acoustic similarity to what they represent

• Analogue spoken elements

  -> used as units of linguistic expression; language users, we may tend to notice those similarities without being fully conscious of them -> being aware of those subtle, continues aspects of spocken language might give some people a minor advantage above the non-analogue spoken language

Example: Onomatopoeia -> like “wauwau” for dog

acoustic language using analogue units

acustic language using non-analogue units

acoustic language using non-analogue units

haptic language using analogue units

analogue static-dynamic acoustics

haptic language using non-analogue units

haptic language using non-analogue units

a