- Frequency Claims: These describe the rate or degree of a single variable (e.g., "45% of students use caffeine daily").
- Association Claims: These argue that one level of a variable is likely linked to a particular level of another (e.g., "Higher caffeine intake is linked to higher GPA").
- Causal Claims: These assert that one variable directly causes changes in another (e.g., "Caffeine consumption increases test scores").
To determine if a claim is valid, researchers evaluate four key validities:
- Construct Validity: Do the operational variables (how you actually measured something) accurately represent the conceptual variables (the abstract idea)?
- External Validity: Can the results be generalized to larger populations, other times, or different situations?
- Statistical Validity: How strong is the effect size, and how precise is the estimate?
- Internal Validity: Can we confidently say that variable A caused the change in variable B, rather than some third variable C?
Experimental Design and Sampling
To establish causality, researchers often use experiments. In these designs, the researcher manipulates an independent variable (IV) to observe its effect on a dependent variable (DV). To ensure the effect is due to the IV, researchers use comparison groups, which might include a treatment group, a control group, or a placebo group.
A critical distinction in research is between random selection and random assignment:
- Random Selection (or Probability Sampling) is used to ensure the sample is representative of the population, allowing for high external validity. Techniques include simple random sampling, systematic sampling, cluster sampling, multistage sampling, and stratified random sampling (which may involve oversampling specific groups).
- Random Assignment is used to ensure that different groups in an experiment are equivalent at the start, which helps establish internal validity by ensuring temporal precedence (the cause comes before the effect) and covariance (the variables change together).
Non-probability sampling methods, such as convenience sampling (using whoever is easy to reach), self-selection (participants volunteering), purposive sampling (targeting specific demographics), snowball sampling (participants recruiting others), or quota sampling, are often easier but may limit the ability to generalize results.
Measurement: Reliability and Scales
How we measure variables determines the quality of our data. We typically use self-report measures (surveys), observational measures (watching behavior), or physiological measures (brain scans, heart rate).
Reliability
A measure is only useful if it is reliable—meaning it is consistent:
- Test-retest reliability: Consistent scores over time.
- Interrater reliability: Consistent scores regardless of who is doing the measuring.
- Internal reliability: Consistent responses within a single measure.
Scales of Measurement
Variables are classified as categorical (labels like nationality) or quantitative (numbers). Quantitative variables fall into three scales:
- Ordinal Scale: Numbers represent a rank order (e.g., 1st, 2nd, 3rd place).
- Interval Scale: Numbers represent equal distances, but there is no "true zero" (e.g., IQ score or temperature).
- Ratio Scale: Numbers represent equal distances and a true zero exists, meaning zero represents "none" (e.g., height or number of correct answers).
Note on Survey Accuracy: When using scales like the Likert scale (e.g., 1-7 agreement) or semantic differential formats, researchers must watch for response sets. These include acquiescence (yea-saying), fence sitting (avoiding extreme answers), and socially desirable responding (faking good or faking bad).
Controlling for Confounds
In any study, researchers must distinguish between systematic variability (changes that follow a pattern related to the IV) and unsystematic variability (random noise). A confound is a type of systematic variability that provides an alternative explanation for the results.
Common threats to validity include:
- Design Confounds: When a second variable varies systematically with the IV.
- Selection Effects: When the groups being compared are not equivalent due to how participants were recruited.
Researchers use different designs to manage these:
- Independent-groups design (between-subjects): Different people in each condition.
- Within-groups design (repeated measures): The same people in all conditions.
While within-groups designs are powerful, they are susceptible to order effects, such as practice effects (getting better because you've done it before) or carryover effects (the first task influencing the second). Researchers may use posttest only designs or pre/posttest designs to help mitigate these issues.
