The missing side of a convincing result

Suppose a learning program surveys its students and announces: “Seventy-five percent support our new schedule!” The calculation might be correct. But what if the survey mainly reached the students who already liked the schedule? What about those who were absent, unable to answer, or never invited?

Lesson 013 introduced falsifiability: we should specify what observations could challenge a claim. Today we ask a companion question: Which observations were actually allowed into our evidence? A hypothesis can be testable and a calculation can be flawless while the sample still misses important parts of the population.

1. The question comes before the sample

Before interpreting a percentage, identify the group the statement is supposed to describe. In survey research this is the target population. The sample is the subset whose observations are included. The sampling frame is the list or procedure by which members of the population could be selected. The respondents are those in the selected or invited group who actually provide answers.

These groups may be different. If the invitation list leaves out night students, the frame may have a coverage problem. If everyone is invited but only some answer, nonresponse becomes relevant. If only enthusiastic learners volunteer, the self-selection mechanism may matter.

2. Selection bias: who had a chance to be counted?

Selection bias can arise when inclusion in the data is systematically related to the characteristic or outcome being estimated. The result may describe the observed group accurately while misrepresenting the group we hoped to understand.

It is not the same thing as the everyday psychological phrase confirmation bias, which describes a tendency to seek or interpret evidence in ways that support existing beliefs. Selection problems can happen without anyone deliberately cherry-picking data.

The American Association for Public Opinion Research (AAPOR) explains the importance of sampling frames and the distinction between probability-based and opt-in samples. The Pew Research Center’s methodology describes coverage, sampling, nonresponse, measurement, and adjustment error as separate concerns.

3. Mini-lab · The vanishing half of the survey

15–20 MINUTES · PAPER ONLY · ALL DATA INVENTED

A fictional Nova Academy workshop considers extending its evening session. All 24 learners are invited to complete a short survey, but only 12 respond. The organizers announce: “75% of students support the change!” You are given a special fictional view of the views of both respondents and nonrespondents, even though a real survey ordinarily does not reveal what its nonrespondents would have said.

Group	Support	Do not support	Total
Respondents	9	3	12
Nonrespondents (fictional hidden answers)	3	9	12
Entire fictional class	12	12	24

Your investigation

1. Calculate the percentage supporting the change among respondents only: 9 out of 12.
2. Calculate the response rate: 12 responses out of 24 invitations.
3. Calculate support among the fictional nonrespondents: 3 out of 12.
4. Calculate the actual support percentage among the full fictional group: 12 out of 24.
5. Compare the respondent percentage with the full-group percentage. How many percentage points apart are they?
6. Rewrite the organizer’s announcement so that it accurately describes the observed data without claiming to know everyone’s views.

4. Check the arithmetic—and the claim

The issue is not that 75% was an incorrect division problem. The mistake would be applying that statistic to everyone without addressing the selection and response process.

In real surveys, researchers do not normally get to peek at the missing group’s true answers. They may compare respondents with reliable external information, follow up with people who did not answer, or use carefully justified adjustments.

5. A low response rate is a warning, not a verdict

In our invented example, the respondents and nonrespondents differ greatly on the actual question, so nonresponse bias is present. But in a real survey, we usually cannot know this from the response rate alone. A survey with many missing responses might have little bias for one outcome, while another survey with fewer missing responses could still be biased if the missing cases differ in a relevant way.

AAPOR’s Standard Definitions distinguishes coverage, measurement, and nonresponse-related error. Assessing possible bias requires more than simply reporting how many invitations received an answer.

Increasing the number of respondents can reduce random sampling variation, but a very large biased sample can still give a misleading answer. Quantity does not replace a sound method of selection.

6. What a fairer sampling plan might do

A more careful investigation starts by defining the group of interest, then selecting or recruiting participants through a plan that tries to give the target population appropriate coverage. When the goal is population estimation, a probability-based sampling approach may be suitable when feasible. But even a probability sample can be affected by nonresponse, bad questions, or recording errors.

For a small class of 24 people, surveying the full defined group may be practical, but that does not force every learner to respond. Nobody should be pressured to disclose personal opinions for a classroom exercise.

7. Beyond surveys: survivorship and visible successes

Selection issues are not limited to questionnaires. Imagine studying only successful projects to discover what makes an idea work. You may overlook all the projects that tried the same methods and failed. Looking only at the surviving examples can lead to a distorted picture of which strategies actually helped.

The same temptation appears in online engagement. Reading dozens of enthusiastic comments beneath a post may accurately describe those commenters; it does not establish what every reader, follower, or nonparticipant thinks. Visible participation is not identical to the entire audience.

8. The sampling-frame audit

5–10 MINUTES · AN OBSERVATION EXERCISE

Find a published poll, a product review page, or an imaginary school questionnaire. Avoid recording private information. Write four lines:

1. Claim: What broad conclusion is being advertised?
2. Counted: Who or what appears in the data?
3. Missing: Who might be excluded, uninvited, unreachable, or unwilling to reply?
4. Revised: What narrower claim is justified without assuming the missing cases behave like the observed cases?

Some opt-in data are valuable for describing those who participated or for discovering new questions. The lesson is not that such data are worthless; it is that population-wide claims require care about what was counted.

9. Knowledge check · Six questions

1. What is the difference between the target population and the observed sample?
2. In the fictional survey, what percentage of respondents supported the change?
3. What percentage of the entire fictional class supported it?
4. Why does a low response rate not automatically prove a survey is biased?
5. How can a sampling frame exclude part of the intended population?
6. Does collecting more observations automatically fix a systematically unrepresentative sample?

---

Reflection · Who isn’t in the picture?

Falsifiability asks whether the evidence could change our claim. Selection bias asks whether we ever gave the challenging evidence a chance to appear. Together they make a more careful kind of inquiry.

Takeaway: Define the population. Examine the selection process. Count who responded. Identify who may be missing. Compare the observed group with what you intend to describe. Report the limits.

Completion: You have completed Lesson 014 when you can reconstruct the 75% and 50% figures, explain the 25-point difference, and write a survey conclusion that names its denominator and limitation.

Sources & further learning

• AAPOR — Best Practices for Survey Research: defining the population, sampling frames, probability and opt-in approaches.
• AAPOR — Standard Definitions: terminology for coverage, nonresponse, and survey errors.
• Pew Research Center — U.S. Survey Methodology: examples of survey coverage, sampling, nonresponse and adjustment considerations.
• Roper Center at Cornell — Transparency in Polling: why response records, definitions, and missing cases matter.

← Review Lesson 013 · Browse the Foundations of Inquiry · Explore the Quantum Atrium

Nova Academy is a free, non-accredited educational initiative of the Electric Icarus Project. The 24-learner survey is entirely fictional. Neither its results nor the model’s hidden answers represent actual Academy student opinions.