CTT Validity

• Does the test/questionnaire appear to test what it aims to test?

• “Face validity” refers to whether the items appear to be valid to the test taker or test user

• Superficial assessment if the measurement procedure used in a study appears to be a valid measure of a given variable

• The extent to which the items reflect a specific domain of content

  • Is the sample of items representative for the domain in question?

    • Example 1: Do items of a math test cover all relevant mathematical operations (addition, subtraction, division, and their combinations)?

    • Example 2: Fitness test should contain elements of strenght, stamina, skill, motor coordination

• Often a matter of judgment – Experts may be asked to rate the relevance and appropriateness of the items or questions

  • E.g., rate each item: very important / nice to know / not important

• Concurrent validity: Degree to which a test corresponds to an external criterion that is known concurrently

  • Two measures in the study are taken at the same time and their scores are measured

    • Example: Correlation between IQ test scores of pupils and their currently obtained math grades

• Predictive validity: Degree to which a test accurately predicts a criterion that will occur in the future

  • Example: Correlation between IQ assed at the age of 15 with grades at the university 7 years later

• Examines the degree to which the operationalization is similar to other operationalizations that it theoretically should be similar to

  • Example: Assessing the convergent validity of a test of arithmetic skills

    • Correlate the scores on a test that is supposed to measure arithmetic ability with scores on other tests that purport to measure basic math ability

    • A strong correlation would be evidence of convergent validity

• Examines the degree to which the operationalization is not similar to other operationalizations that it theoretically should be not be similar to

  • Examples: Assessing the discriminant validity of a test of arithmetic skills:

    • Correlate the scores of a test which is supposed to measure arithmetic ability with scores of tests of verbal ability, where low correlations would be evidence of discriminant validity

    • Average correlation between IQ tests and tests of attention: r≈.20

• Evidence for discriminant validity: Low or zero correlation

  • Discriminant correlations should be at least smaller than convergent correlations

• Showing that two tests are negatively correlated does not imply discriminant validity

  • Because they are, in fact, related and we could use one test to predict scores of the other tests

Example: Correlation between an extraversion and an introversion scale: r = -.90, put differently, r2 = .81

• 81% shared variance

• It is hard to distinguish between these constructs

• Introversion is just the opposite of extraversion

• This is actually an example of convergent validity!