Discuss the following statement: “Multimodal systems are radically different than standard GUIs” (Core elements of GUI, Challenges)
Multimodal Systems aren’t enterely different than GUIs -> they build on GUI principles, but their paradigm shift lies in breaking the dependency on screens and hands.
They expand how users can interact by allowing natural, flexible, and inclusive input/output.
Core Element of GUI:
windows, icons, menu, pointers > WIMP model for GUI
not every element has to be a part
Challenges:
multimodal interaction is generalization of GUI-based interaction, but GUI-based interatction is multimodal es well
there are far more diverse, multimodal and blending-with-people systems -> lead to new usability issues
-> GUI are not becoming obsolete
-> GUI-based interaction is a usegul multimodal interaction paradigm
came first historically -> better explored and more familiar to most people than other kinds of multimodal interaction
Explain the five medium / carrier / sense correspondences
Precisely define the term “modality” (using your own words)
combination of a physical signal (e.g. sound) and the human sensory system (e.g. hearing) used to exchange information through a medium
It defines how an user gives input or receives feedback
many & different modalities can be used for representing information in the same medium
Why do we have to differentiate between input and output modalities?
to give a precise definition of the system at hand
(e.g. spoken computer game >> is it a game controlled by speech? with a speech output? with speech input and output?)
Which modalities are used by interfaces which you use day by day (e. g. smart phone, computer, smart watch, smart TV, ticket terminal, etc.)
Output modalities:
Graphics: screens (e.g. smart phone), text, images
Haptics: vibration (e.g. apple pay)
Acoustics: Sounds (e.g. apple pay)
Input modalities:
Haptics: touchscreen, typing on a keyboard, using a mouse
Acoustics: voice input (e.g. Siri)
Graphic: FaceID
Other approach:
Linguistic: reading text, writing on keyboard / touchscreen
Analogue:
Arbitrary:
Explain the two major approaches for categorizing different modalities. Which one is more intuitive for you?
based on the medium or senses modality grouping (graphic, acoustic, haptic)
based on structural modality type (linguistic, analogue, arbitraty, structure)
>> the first because it starts with the sensory experience so it feels more like thinking of the user first and not of the system modelling
Please categroize your “day by day modalities” by one of the two approaches.
reading a text message: graphic modality >> static graphic language >> static text (5a) >> typed text (5a1)
typing a message: graphic modality >> dynamic graphic language >> dynamic text (8a) >> typed text (5a1)
listening to podacast: acoustic modality >> acoustic language >> spoken discourse (6a)
using emojis: arbitrary static graphics
Define multimodal system
multimodal system has both input and output
uses at least two different modalities for input and / or output
( unimodal system: has input and output & uses the same modalitie for input and output)
man in video uses voice input to create circles, triangles and squares on screen (“create a …”)
he points to the direction where he wants the element created (”here”)
he moves the elements by voice input and pointing -> element gets selected (“move that”) and then moved to desired location (“there”)
also moves elements without gesture by giving the location via voice
Voice (acustic)
pointing gesture (haptic)
show elements on screen (visual)
show selected elements in a different color (visual)
Zuletzt geändertvor einem Monat
