Nova Academy — Lesson 019: Sampling Variation — Why Good Samples Disagree

Nova Academy · Foundations of Inquiry · Lesson 019 · November 8, 2026

Learning objectives

Explain what a sampling distribution represents.
Distinguish sample variation from measurement error and bias.
Calculate a sample mean and describe why estimates differ across samples.
Explain how sample size affects the variability of estimates, without assuming larger samples eliminate bias.

From random assignment to sampling uncertainty

Lesson 018 examined random assignment: using chance to allocate participants to experimental groups. Today we ask a different question: if we randomly select people or observations from a population, how much might our sample result vary just because we happened to select different individuals? Random sampling concerns who enters a study; random assignment concerns which treatment a participant receives. Neither automatically guarantees the other.

A sampling distribution is the distribution of a statistic (such as the sample mean) across many hypothetical repeated samples of the same size from the same population using the same sampling method. We ordinarily observe just one sample; the sampling distribution helps us understand the uncertainty in its statistic.

Worked example: four equally likely observations

Suppose a tiny population consists of four values: 2, 4, 6 and 8. Its population mean is (2+4+6+8)/4 = 5. Draw two values independently with replacement, so there are 4 × 4 = 16 equally likely ordered samples. Their sample means are:

First draw	Second draw: 2	4	6	8
2	2	3	4	5
4	3	4	5	6
6	4	5	6	7
8	5	6	7	8

The sample means range from 2 to 8, even though the true population mean is always 5. Across the 16 equally likely samples, the average of the sample means is exactly 5. The population variance is [(2−5)²+(4−5)²+(6−5)²+(8−5)²]/4 = 5. Because we draw independently with replacement, the variance of a two-draw sample mean is 5/2 = 2.5; its standard deviation (the standard error) is √2.5 ≈ 1.58. For four independent draws, it would be √(5/4) ≈ 1.12. These are population-based theoretical values, not estimates from a single real sample.

Hands-on activity

Write 2, 4, 6 and 8 on four slips. Draw one, record it, replace it, mix, then draw again. Calculate the two-number mean. Repeat for 20 two-draw samples. Make a tally chart of the 20 means and compare their distribution with the table above. Then try 20 four-draw samples. Are the four-draw means generally less spread out? Chance may produce exceptions in only 20 trials.

Interpretation and limits

With independent random sampling, larger sample sizes usually reduce the random variation of sample means, roughly in proportion to 1/√n when the population variance is finite. But large samples cannot repair a biased selection process, poorly defined measurements, nonresponse, or confounding. Sampling without replacement changes the exact variance, especially when the sample is a large fraction of a small population. These toy values are deliberately simple; real research often needs appropriate statistical models, uncertainty intervals, and careful sampling design.

Reflection

If two honest surveys disagree, must one be wrong? What questions about sampling and measurement should you ask before comparing them?

Knowledge check

What does a sampling distribution describe?
What is the population mean of 2, 4, 6 and 8?
How many ordered two-draw samples with replacement are possible?
What is the sample mean if you draw 2 and 8?
Does a larger sample automatically eliminate selection bias?
How does random assignment differ from random sampling?

Answer key

The possible values and frequencies/probabilities of a statistic across repeated samples under a specified sampling design.
5.
16.
(2+8)/2 = 5.
No. Bias can persist or become more precisely estimated.
Random assignment allocates sampled participants to conditions; random sampling selects observations from a population.