Unit 2: Survey Design and Construction

Element

An object on which a measurement is taken

Population

A collection of elements about which we wish to make an inference.

Sampling Unit

Nonoverlapping collections of elements from the population that cover the entire population.

(Sampling) Frame

A list of sampling units

Sample

A collection of sampling units drawn from a frame or frames.

A unit and an element may or may not be the same thing.

What could your sampling units be instead so that your elements and sampling units would not be the same?

List of addresses/households

--> but you actually are surveying individuals

--> elements would be the people who live there but the company only has the addresses

Suppose I wanted to survey adults in the City of Winnipeg about how often they recycle. Our elements would be adults and our population would be adults in the City of Winnipeg.

• What might we use for sampling units?

• What might we use for a sampling frame?

Sampling units: individuals, households

Sampling frame: census, record of taxpayers, list of telephone numbers, list of registered voters

Sample size

What sample size should we use?

We estimate a population parameter with our best guess, a sample statistic.

Every sample is different but ideally we’d like to make sure our estimate is within a certain amount, say B.

Sampling Method

How do we choose a sampling method?

This depends largely on the context of the study so we must ask ourselves:

How are our elements spread out throughout our sampling frame? How easy is it for me to access them?

• Are elements easy to sample with no groupings of common elements —> simple random sampling

• Are there defined subgroups? —> stratified sampling

• Are our sampling units very geographically spread apart?—> multistage sampling.

• Are there defined groups where there’s no big difference between groups but inside the groups it is diverse? —> cluster sampling

• Do I have an easily accessible list where the order is more or less random? —> systematic sampling

Errors of nonobservation are related to our sample making up only part of the target population.

Examples:

• Sampling Error

• Undercoverage

• Nonresponse

Errors of Non-Observation: Sampling Error

• Sampling Error: The distance between the recorded statistic and the population parameter due to only collecting a sample of the population.

• Example: our statistic changes between each sample merely because each sample is different, not because the parameter is changing

• Sampling error is something we have to live with as the price of being statisticians —> we can at least control this by setting α

Choice of Significance Level α

We can choose α greater than or smaller than the standard 0.05 if it is suitable for the study.

What are some scenarios in which a small α may be preferred? What about a larger α?

• Small alpha: medical trials --> avoid bigger mistakes

• Large alpha: get a general overview for the model and accuracy is not that important as before

• Important: setting alpha before analysis and not depending on the p-value just to have the significance of your results

Errors of Non-Observation: Undercoverage

• Undercoverage: When a sampling frame does not include the entire target population.

• Note: There can also be issues with the sampling frame containing units not in the target population, e.g., Non-Canadians.

• Correction: Issues with coverage are hard to correct after the fact as there is usually a reason they were not included in the original sampling frame. Responsibly, you should report what your sampling frame was and how it compares to your target population in your study

Errors of Non-Observation: Undercoverage

Question: What are some examples of scenarios where your sampling frame may be missing elements from the population?

Answer: List of telephone numbers for individuals but not everybody has a number --> if significant individual does not have a phone number this can lead to undercoverage

Question: What about where your sampling frame contains elements not in the target population?

Answer: Asking people on the street questions even though they are actually not Canadians

Often those missing, are missing for reasons that may make them important and unique parts of your population to survey. In particular, vulnerable or low income populations can be marginalized from participation in opinion polling.

Errors of Non-Observation: Nonresponse

• Nonresponse: When you cannot collect measurement on selected units in your sample

• Example: e.g. when people do not respond to mail, on phone, ...

Errors of Non-Observation: Nonresponse

We can broadly classify non-response into three causes

• An inability to physically reach a sampling unit, e.g., no internet connection, no phone line, no permanent address.

• An inability of the sampling unit to give the correct response.

• A person may refuse to answer the survey because, e.g.,

  • Questions where you do not feel comfortable to answer

  • Survey is too long and would take too much time

  • Not interested in doing so

    
    

Errors of observation are related to what is recorded about our sampling units being inaccurate.

We can broadly group errors of observation as being due to:

• Interviewers: Tone, age, gender, physical appearance, and demeanor of an interviewer can all affect how truthful people will be, intentionally or unintentionally

• Respondents: Respondents might not understand questions, may not seek clarification, may be embarrassed or afraid to answer truthfully, may exaggerate, may make up answers to not appear uninformed, or confuse units of measurement.

• Measurement Instrument: Confusion around what the unit of measurement is or how something is defined.

• Method of Data Collections: Accuracy can be affected by conducting personal interviews vs telephone interviews vs self-administered questionaires vs direct observation.

Personal interviews vs. telephone interviews vs. mailed questionaires

Personal interviews

• non-response may be lower with personal interviews (as people my not have time for a personal meeting)

• could result in more detailed responses

• may get biased responses from trying to agree with the interviewer

Telephone interviews

• people may be more willing to partiticipate

• could result in more detailed responses

• may get biased responses from trying to agree with the interviewer

Mailed questionnaire

• participatnts may be more honest

• non-response may be higher unable to clarify question

Reducing Error

There are many ways research companies and researches attempt to reduce errors in their surveys:

• Callbacks: Making follow-up calls or sending reminder surveys (by mail or email) can help response rates. Follow-up calls should vary by time of day and week to catch people on different schedules.

• Rewards and Incentives: Surveys can offer monetary incentives for participating or put respondents into draws for a potential reward. Large survey companies with panels of people they select from may earn points towards gift cards or other rewards.

• Interviewer Training: Interviewers should have opportunities to practice asking questions under watchful eyes that can suggest improvements in intonation or pronunciation or demeanor that may get more truthful answers.

• Data Checks: Data can be cross referenced (e.g. age to year of birth), obvious “wrong” answers can be eliminated or corrected by followups if possible.

• Questionnaire Construction: Questions can be constructed to help get honest and truthful answers from respondents and help eliminate people from lying due to not understanding questions.

Question Ordering: Primacy Effect

• Primacy effect: make people more likely to select the first choice on a list or stop reading when they get to their first agreeable option.

• Example: In an online survey asking participants which food item they consume the most from a list of 30+ foods, items near the top of the list may get selected more than those at the bottom.

  --> in long lists respondents tend to select itmes in the beginning of the list

  --> try to categorize and make multiple questions

Question Ordering: Recency Effect

• Recency effect: makes people more likely to choose from the last few options on a list because they remember them

• Example: In a telephone survey asking participants to choose the most important political issue to them, more responses may be received for the options at the end of the list because they remember them the best.

  —> avoid long lists

• 
  

Question Ordering: Context Effect

• Context effects are particularly common if one goes from specific question to a more general one or vice versa.

Example 1

• People were asked if they were happy in their marriage and if they were happy with their life in general.

• When asked about life then marriage, 52% said they were very happy in life in general.

• When asked marriage then life, 38% said they were very happy in life in general.

• The theory being people who felt happy thinking about their marriages specifically, that made life in general seem less great in comparison.

Example 2

• A person that is given a long list of questions about crimes might respond differently to a question about if they’ve been a victim of crime.

• The first question “primes” them and gives them opportunities to remember things that have happened to them in the past.

Example 3:

• Respondents may be asked the following

  • A: Will you support an increase in taxes for education?

  • B: Will you support an increase in taxes?

• If A were asked first, those who support a tax increase for education and respond with “Yes” may think that B implies an increase in taxes not for education and respond with “No”.

• If B were asked first, those same people may respond “Yes” to both.

• Note: General question first, then the specific one.

  
  

Question Ordering: Combat unintended effects

Solutions to combat unintended effects include:

• Randomization amongst all participants

  1) randomize question order (when it makes sense)

  2) for lists: randomize order of options

  3) for ordinal questions: flip order of options, e.g., not satisfied ... highly satisfied vs. highly satistified ... not satisfied

• Write out or restate questions

  —> help participants refocus on the given question.

Close questions

• Closed questions: have a predetermined set of answers or a finite numerical answer (e.g. age or pain rating 1 to 10)

• Pros: easy to analyze, may be easier for participant

• Cons: difficult to capture all possible choices, responses may be biased based on order of choices, some categories may get chosen more than would be in an open question

Open questions

• Open questions: allows people answer however they would like

• Pros: allows respondent to give detailed response, may receive responses that would not be on a closed list and that they didn’t even think of

• Cons: difficult to analyze since respondents don’t have a specific format, difficult to compare across questionnaires

Pre-surveys

• Sometimes a pre-survey is done to get the most common options for the real survey.

• This helps to capture what the public will answer as opposed to what the surveyers think they might answer.

Response Options: Forced Choice

• Forced choice questions: makes a respondent select a yes or a no, a one or the other.

• Forcing people to make decisions on questions they know nothing about is not valuable

• Questions about which everyone has enough information on to form an opinion should be stated without a “No Opinion” or “Don’t Know” option

• Fewer options may be given as an attempt to polarize opinion on one side or the other

• Example: Consider the following question: “Do you think the enforcement of traffic laws in our city is too strict or too lenient?” —> No middle choice is given as respondents may tend to choose it too often as an easy way out.

• In certain situations it seems highly likely that people are more accurate when doing forced choice rather than “select all that apply”. The select-all-that-apply format yielded a large primacy effect, while the forced-choice format showed no primacy effect.

• Example: Someone is unlikely to report they’ve suffered from addiction when they haven’t. However, someone may report they haven’t suffered from addiction when they have (to not embarrass themselves) so the reporting method with higher results is likely more accurate here.

Question Wording: Leading Questions

• Leading questions: includes extra information that purposefully influences people in a particular question.

• Example

  • When people were asked “Do you think the United States should forbid public speeches against democracy?”, 21.4% said yes

  • When they were asked “Do you think the United States should allow public speeches against democracy?”, 47.8% said no

  • Forbid is a strong word, allow is much milder

    —> the outcome can be misrepresented depending on the wording

• It is generally good to give someone two options in the wording rather than a straight “Do you favour. . . ?”

• Example: “Do you favor or oppose the use of capital punishment?” over “Do you favor capital punishment?”

• Asking “Do you agree with. . . ?” may make the interviewee feel like the interviewer thinks it’s agreeable and make them more likely to respond with yes

Question Wording: Double Barreled Question

• Double barreled question: a question that addresses to two ideas

• Only one question should be asked at a time.

• Example: “Do you believe the IB program helped promote thinking about global issues and multiculturalism?”

  —> people could have different opinions on this

Question Wording: Double Negatives

• Don’t use double negatives

• Example: “Do you favour or oppose not allowing teenage drivers to drive alone after midnight?”

Question Wording: Measurement Instruments

• Be clear in writing out the measurement instruments for questions with quantitative responses

• Example: “How much do you work a week?” could be better as “On average, how many hours a week are paid for work?”

• For in person interviews, a prop might be helpful to demonstrate height or volume

• Hospitals often give pain scales with descriptors for each number when asking questions of patients