Vorlesungsfolien

Evaluation components:

• setting

• evaluation time

• evaluation partner

• result type

Clear structure:

• Introduction

• Warm up

• Main body

• Closur

occasionally anonymity cannot be guaranteed,

Study method that catches results that are difficult to catch otherwise

Idea: While sombeody is doing something (task), he is constantly explaining what he’s thinking. They are alone and have to handle the system (you are not allowed to give hints,..)

Candidates: preferrably non-experts with comparable verbalization skills

Prepare: Explain system, setting and expectation

Perform: Record (stay out of sight)

Finalize: Extensive interpretation

1. Preparation

2. Perform the Study

3. Finalize

• oftentimes qualitative and subjective evaluation since generalizations are very difficult

• interpretation can be done on various psychological theories and models

-> improve session by:

- look for issues, categorize obvious issues

Study method to measure the mental workload of a system. How?

Using following criteria: (DR UBSS)

• Sensitivity

• Selectivity -

• Bandwidth

• (Un)Obstrusiveness

• Reliability (Reproducability)

• Diagnosticity

e.g. using Microsoft Word for ages but don’t know all the features

1. primary task - directly correlated with the task (usually get an absolute value): measure time, speed, strength and derive workload metrics

2. secondary task - indirect: while task, person has to do rythmic tapping or give random numbers (tapping variability/randomness decreases with workload)

3. physiological corrolates - measure physiologic metrics like heartrate, problem: conditioning required! comparison between subjects, difficult setup

   to achieve Conditioning:

   1. baseline phase (relax to get baselevel),

   2. interaction phase (do task),

   3. recovery phase (recovery to base level)

4. subjective ratings - apply popular workload scales like NASA Task Load indeX

Index to measure percieved mental workload (subjective)

Two Parts:

1. 6 Dimensions: Mental, Physical, Temporal, Performace, Effort,

   Frustration (TEMPPF?)

2. Compare dimensions: Score 0-100, optimum score is determined by case

   how?

   1. for each dimension, sum the number of times it was selected as the leading pair wise factor,

   2. Divide this sum by 15 (the number of pairwise comparisons)

   3. Multiply this result by the score for each dimension

   4. Sum the resulting dimension scores

Study method, which closely observates the user

Goal: capture participant experience and it’s context

Essential Parts: (PPT)

• The person: Who?

• The place: Where?

• The thing: What?

(More precise parts: Space, Actors, Activities, Objects, Acts, Events, Time Goals, Feelings)

Main benefit:

user cannot hide -> take out user bias

(normally done in early stages of project)

Goal: capture participant experience and it’s context

• direct observation

  Think aloud

• indirect observation (tracking user’s activities)

  Diaries, Interaction logs, web analytics

• Both (either or)

  Video, audio, photos, notes

Contextual Inquiries

while ethnographic observation: few questions related to the context

nowadays popular (technology enables it easily)

Broad Difference:

Field Study -> you go to the participant, uncontrolled environment

Lab Study -> participant comes to you, controlled environment

• INTERVIEW AND QUESTIONNAIRE

• THINK ALOUD

• OBSERVATION AND ETHNOGRAPHY

• FIELD STUDIES

Use of XY in everyday lives is difficult to do in lab studies, so we are using field studies (e.g. living labs)

field studies can complment lab studies

- In situ (natural habitat) Field studies -> uncontrolled environment

- more likely to find novel insights

- expensive

Difference to Usability Lab studies?

- additional effort:

need to know which conditions to control -> pre studies

need to know which participants…

I. studies of current behavior (what are people doing now?)

-> Result: better understanding of current use of technology

-> Study: open ended questions

-> “How do family members share technology (TV)?”

II. proof of concept studies (does my technology function in the real world?)

-> Result: validation of a new technology

-> Study: Give out technology, specific questions

-> “Give out phone App (+ ask: does it work?)”

III. experience studies (How does my prototype change people’s behavior?)

-> Result: Experience with a new product

-> Study: Experience with using a technology for a longer period of time, Wizard of Oz

-> Experience in the long run

Surveys

-> regular survey after any change of condition

Logging

-> use mobile device to also collect data about usage

Used to conduct a Experience Study (Ubicomp field study type III)

= simulates and controls system from behind the scenes (use mock interface and interact with user

(appear as a smooth, already implemented system)

• good for proof of concept

• expensive

Questionnaires at various point throughout the day/event/… (via smartphone)

Why? Records changes in users behavior when using an user interface, not only the usability of the interface

Grouping (professionals, non professionals)

-> based on expertise, frequency of use, demographics

Data sampling strategy

-> random sampling,

-> systematic sampling,

-> stratified sampling,

-> samples of convenience

Representation of Study Participants (to the intended user group)

-> representative vs. non-representative participant set

depends on what you are looking for:

Jakob Nielsen recommends: rather few participants but frequent studies! (5 users identify 80%) -> experimental study idea)

but: validity depends on statistical strength

therefore it depends on acceptable error

• probability based study of behavior -> z-Test: choose confidence level and error margin: N = (0.25 z^2) / w^2

• controlled survey -> controlled sample size (e.g. sample of convenience)

• experimental study -> 5 evaluators identify 80% of usability issues

• Within Subjects (one participant gets several tasks, result compared to other tasks)

• Between Subjects (everybody gets same task, compare how different users perform the task) users have to be grouped

• mixed

If you have several options: teest order may influence the outcome

-> randomize (using a Latin Square)

1. concrete research question

2. Answer the questions:

   a. Screen (Who?)

   b. Schedule (When?)

   c. Setup (What will they do?) -> short and precise!

   d. Data (What data to collect?)

=> put everything in a design document for the study

(Research questions, Participant profile, Method description, Timeline, Data to collect, Analysis, Validate Hypothesis, Incentives)

Quantiative and Qualitative (= mixed) offers maximum insight

Define dependent and independent variables:

- independent: what you vary to measure an effect (different types of haptic stimulation on the glove)

- dependent: effect to measure (how many errors they do when playing the piano)

Dependent/Independent variables can be influenced by:

- confounding variables: influences both independent and dependent variables (pre-knowledge about the piano piece)

- extraneous variables: only influences dependent variables (environment is 35° and you can’t see anymore)

How to solve?

- randomize

- matching

- statistical control

- design control

• Depends on type of study (experience studies longer than behaviour study)

• depends on novelty (entirely new system?)

• pracical considerations (effort from participants, breaks)

• frequency of use

VL 2 Slide 160 & 162 (p. 146 & 148)

• Reliability: does the method produce the same results on separate occasions?

• Validity: does the method measure what it is intended to measure?

  • Internal Validity (certainty that you know the cause)

  • External Validity (result can be generalized)

  • Ecological validity: does the environment of the evaluation distort the results? Is the result transferable to a general environment?

• Biases: Are there biases that distort the results?

• Scope: How generalizable are the results?

Inspect study without the user: have domain experts look over the study

Inspection:

• use multiple Inspection methods (several kinds)

• review with domain experts

• use heuristic evaluation

• walkthroughs

What should experts look at: ask them about

• match between system and real world

• user control and freedom

• consistency

• error prevention

• …

Both are examples for expert evaluation techniques of a study:

1. Cognitive Walkthroughs:

   Designer presents design, usage scenario and context

   -> Domain expert tries to imagine a user doing it and is guided by questions of the designer: Will the user notice the correct action?

2. Pluralistic Walkthrough:

   multiple experts do a cognitive walkthrough, discuss afterwards

Human: Input, Processing, Output,

Computer: Input, Processing, Outut (same)

(“both Outputs are very interesting, interpretation of output is up to you”)

Senses -> initial Analysis -> Memory -> Motor Systems

-> Output from speech/muscles…

(SIMMO)

Stimuli on left, Responses on right, inbetween different streams (parallel, unparallel,…).

Blue arrow is attention ressources (Information Processing Ressources), limited

Senses as “windows to the world”

A system that responds to physical energy and that corresponds to a defined region within the brain where signals are recieved and interpreted

Sensory information,

Sensory stores (Iconic, Echoic = Vision, Hearing)

Sensory input that is selectively attended

Working Memory -> Output, Store in / Retrieve from Longterm meomery (Articulatory Loop)

can store perception after the stimulus,

• echoic memory (2-10 sec after stimulus, hearing)

• iconic memory (0,5-1 sec after stimulus, visual)

tightly related to attention ressources

Attention: Cognitive process of selectively concentrating on one aspect of the environment while ignoring other things

-> can be directed to a particular task or divided between different tasks

-> practice reduces amount of attention required by a particular task

Types of attention:

• selective attention (willingly select focus, one source)

• focused attention (respond to external events, not willingly, e.g. somebody talks nearby, one source)

• divided attention (simultaneous focusing on different events, multiple sources)

Optimum: selective attention

Allocation model:

- limited amount of “processing power” at our disposal

- task execution depends on how much of our attention “capacity” we can spare on it

-> controlled processing (heavy attention demand)

-> automatic processing (takes no resources)

=> practice reduces amount of attention required by a particular task

“Vigilance is an aspect of attention which refers to detecting a rare event or signal in a desert of inactivity or noise” (Wachsamkeit)

Types:

• time

  • free-response paradigm: can occur anytime (e.g. flight simulator training)

  • inspection paradigm: regular small errors (e.g. quality control)

• order

  • successive vigilance paradigm: has to be remembered (e.g. color darker than initial target)

  • simultaneous vigilance paradigm: all information present, compare

• perception

  • sensory vigilance paradigm: signals change intensity (e.g. color change)

  • cognitive vigilance paradigm: signal represents information

two stages of information processing in the task of detection

1. sensory evidence is gathered (= recognition)

2. decision is made

decision can be influenced by noise:

Types of noises

• external (in the data, snow overlaying sign)

• internal (in our brain, same stimulus different neuronal activity)

Internal Response = neural activity resulting from a stimuli

evaluated by

• decision criteria Xc

• response bias (beta, rarely adapted by humans -> “sluggish beta”)

Sensitivity: resolution of the detection mechanism

= separation of noise and signal (distribution along x axis)

low sensitivity: both kind of same percieved

high sensitivity: clearly distinguishable noise and signal

working memory and long term memory

Memory Processes:

• Recall

• Recognition

• Chunking

• Rehearsal

=> Multi Modal

Working Memory:

• 30 seconds storage

• 70ms access times

• storage size: 3-4 chunks,

• visual (visuo-spatial sketchpad) or verbal information (articulatory loop)

• chunk refers to a unit of information that the brain can hold as a single entity

• easy to overwrite

• easy trigger: sound

Types

• Verbal: Phonological Store, Articulatory Loop

• Visual: Visuospatial Sketchpad (“Klangbilder”)

• Operate in parallel

LTM:

• “unlimited” in capacity, from a few minutes to life time

• slow access (100ms)

• triggers: smell

Three types:

• episodic

• procedural

• semantic

Encoding - information is stored in memory

Retrieval - memories are recovered from long-term storage

Forgetting - fail to recover information

Recall - retrieve particular piece of information

Recognition - (“Wiedererkennung”) search memory and decide whether retrieved piece matches info

Rehearsal - repeating information in working memory facilitates transfer from WM to LTM

Designer helps you chunking

decay: information lost gradually, slowly

interference:

• retroactive (new information replaces old)

• proactive (old info interferes with new)

Remember over long period of time difficult

-> Recognition might be of help:

• much easier to recognize than recall something

• hold images and texts

Web Pages difficult to recall because they change format -> Recall often also bound to shape

=> How to implement? Icons very useful since

• use metaphor to recall

• understandable, familiar, memorable, informative,…

Guidline: practical rules for solving UI design problems

Principle: help for analyzing design alternatives

Theories: describte objects and give explanation of connections

• Organize in Chunks

• Short linear sequence of tasks

• Persistence (do not flash info shortly)

• Reminders

• Consistent user interface

• Consistent Icons/Colours, Terminology

Decision: After perception of stimulus, response needs to be selected

• automatic: fast, little attention required, reflexes or behavior from LTM

• controlled: slow, attention required, interaction with WM and LTM

Response: after decision made, it is executed by complex motor movements

Feedback: observe consequence of actions, producing feedback

(circular model)

Most influential model of user interaction: wants to quantify numbers on how long the human processing takes

-> These numbers are used later for the Interaction design models

consists of 3 interacting subsystems:

• Perceptual Processor (sensory inputm output to audio/visual storage)

• Cognitive Processor (access LTM to determine response)

• Motor Processor (carry out response)

principles of operation

• Fitts Law,

• Power Law of Practice

Serial Action or parallel perception

all about Processor Cycle time

Power Law of Practice

How often do we have to percieve an activity before we know it and perform faster?

-> task time follows a power of law:

Tn = T1 * n^-0,4

=> you get faster every time, perceptually and motorically (but not knowledge acquisition)