Nova Academy — Lesson 021: Experimental Design — How to Test a Question Fairly

Foundations of Inquiry · Lesson 021

Experimental Design: How to Test a Question Fairly

Learning objectives: Identify a research question, outcome, treatment, comparison group, confounder, and random assignment; explain how blinding and replication strengthen an experiment; distinguish causal evidence from an uncontrolled observation.

1. A question becomes a test

Suppose a school wants to know whether a ten-minute daily practice quiz improves test scores. An experiment should specify who participates, what changes, what is measured, and how the groups are compared. The treatment is the practice quiz; the outcome is the score on the same final assessment; a comparison group follows the usual routine.

2. Fair comparison

Randomly assign consenting participants to the quiz and comparison groups when practical and ethical. Random assignment helps balance both known and unknown differences on average, but it does not guarantee perfect balance in any single experiment. Keep instruction time, test conditions, and scoring as comparable as possible. If students choose their own groups, motivation may confound the comparison. Blinding graders to group assignments can reduce scoring bias.

3. Worked example

In a hypothetical randomized trial, 20 students receive practice quizzes and average 78 points; 20 comparison students average 72. The observed difference in sample means is 78 − 72 = 6 points. This is a descriptive estimate, not proof that the true causal effect is six points. Without individual-level variation, uncertainty intervals and significance tests cannot be calculated responsibly. A larger, well-designed replication would help evaluate precision and generalizability.

4. Hands-on activity

  1. Write a testable question about two ways of studying or organizing materials.
  2. Define one measurable outcome and the time frame.
  3. Describe a fair comparison group and how assignment would work.
  4. List two potential confounders and one practical way to limit each.
  5. State what result would challenge your prediction, not just confirm it.

5. Limitations and ethics

Small groups can yield noisy results. Dropouts, noncompliance, selective reporting, measurement errors, and unrepresentative recruitment can limit conclusions. Random assignment is not always possible or ethical; in those settings, observational designs require more cautious causal claims. Obtain appropriate consent and protect participant privacy.

6. Reflection

Why might two groups have different average outcomes even when the treatment has no real effect? How could the same experiment be repeated by another investigator?

7. Knowledge check

  1. What is the independent variable in the practice-quiz example?
  2. What is the outcome?
  3. Why randomize group assignment?
  4. What does the six-point difference establish?
  5. Name one limitation that random assignment does not automatically solve.

Answer key

  1. Whether students receive the daily practice quiz.
  2. The final assessment score.
  3. To reduce systematic baseline differences between groups on average.
  4. Only that the observed sample means differ by six points; uncertainty and causation require further design and analysis considerations.
  5. Examples: attrition, measurement bias, small-sample variation, or poor generalizability.

Sources and further reading

Review Lesson 020 — Confidence Intervals · Nova Academy index