A finding becomes a question for the next investigator

In Lesson 011, our imaginary paper-airplane comparison found that Design B traveled farther on average than Design A. But the comparison contained only five throws per design, and the figures were invented for teaching. Before turning that difference into a confident generalization, we need to ask what happens when the question is tested again.

A good investigation leaves enough of a trail that another learner can check the arithmetic, understand the methods, and gather fresh observations. These are related but different acts. Our aim is not to defend an exciting first answer at all costs. It is to find out how well the answer survives scrutiny.

1. Two kinds of returning

The National Academies of Sciences, Engineering, and Medicine use these specific definitions in their 2019 report Reproducibility and Replicability in Science:

Disciplines sometimes use these words differently, so good communication specifies the intended meaning. Reproducing a calculation is not the same as independently repeating an experiment. Replication does not require exactly matching every measured value.

2. Why a result may change when you try again

Repeated studies may differ because of ordinary sampling variability, measurement error, differences in procedure, changing conditions, publication biases, incomplete descriptions, or a mistaken causal model. Disagreement can expose a limitation in the original study, but it does not automatically establish dishonesty or misconduct.

Likewise, a consistent numerical result does not guarantee that the original analysis was correct; repeating the same faulty calculation can reproduce the same mistake.

The National Academies report explains why both transparency and the interpretation of replication attempts matter.

3. Mini-lab · Return to the paper airplanes

15–20 MINUTES · PAPER ONLY · ALL DATA INVENTED

In Lesson 011 we compared two hypothetical airplane designs. We now imagine a second round with five new throws per design. The numbers below are invented classroom measurements, not an experiment we actually performed.

Round (5 throws/design)	Design A mean	Design B mean	B − A
First five throws	3.12 m	3.30 m	+0.18 m
Second five throws	3.28 m	3.14 m	−0.14 m
Combined ten throws	3.20 m	3.22 m	+0.02 m

The first round gives B a 0.18-meter lead. The second gives A a 0.14-meter lead. When both equally sized rounds are combined, B leads by only 0.02 meters. This is not proof that the designs are equivalent—or that B really has a positive causal effect. The samples are small and we have not estimated uncertainty.

Try the calculations yourself

1. Subtract the mean of A from the mean of B in the first round.
2. Repeat for the second round, noting the change in sign.
3. Calculate the combined mean for each design. Because each round contains five throws, average the two round means equally.
4. Compare the combined difference with the original difference.
5. Write one sentence describing the pattern and one stating what remains uncertain.

4. Reproduce the numbers from individual observations

A transparent investigation lets another reader verify the arithmetic, rather than trusting only a summary table. Below is one possible set of invented second-round measurements that produces the displayed averages:

Design	Second-round distances (meters)	Sum	Mean
A	3.2, 3.3, 3.4, 3.1, 3.4	16.4	3.28
B	3.0, 3.1, 3.2, 3.1, 3.3	15.7	3.14

Add the five values for each row and divide by five. A classmate who can obtain the same means from these same invented values has reproduced the computation in a small educational sense. They have not collected new flight data, so they have not independently replicated the experiment.

5. Write a procedure someone else could actually follow

An instruction such as “throw a paper airplane normally” leaves many critical decisions unstated. An independent tester needs enough detail to understand how the experiment was conducted without guessing.

Safety matters more than experimental neatness: use a harmless paper model in an uncluttered space. If physical testing is inconvenient, remain with the paper exercise.

6. A new result is not a demand for an identical number

Every measurement comes with variation and limits. In many scientific settings, consistent replication means findings are compatible given their uncertainty, not that all results match to the last decimal place.

When results disagree, check the research questions, measurements, procedures, statistical uncertainty, and possible conditions on which an effect depends. A result that holds in one environment and not another may identify an important boundary condition, not merely a failed investigation.

With our two five-throw samples, we have not calculated a standard deviation, confidence interval, or statistical model of the difference. We should therefore avoid claiming demonstrated superiority or demonstrated equivalence.

7. Replication, publication, and intellectual continuity

Publishing a finding makes it available for scrutiny; it does not make it permanent truth. Good research records its steps clearly and welcomes independent assessment. A stronger conclusion may emerge from confirmation, careful qualification, or recognition that the first estimate was unstable.

For Nova Academy, this becomes a learning principle: the continuity of inquiry lives in the question, the method, and the record—even when the answer changes. This is an educational and philosophical interpretation, not a separate experimental law.

The Quantum Atrium provides a neighboring pathway into deeper research and source-based exploration as the Academy expands.

8. Field note · Ask a second investigator

Do not assume a second attempt must agree with the first. Ask what the disagreement teaches about the method, the setting, and the original claim.

9. Knowledge check · Six questions

1. Does repeating the same spreadsheet formula on the same data collect new evidence?
2. What is replicability in the National Academies’ specific definition?
3. Why does the second airplane round change the simple story told by the first?
4. What is the combined B-minus-A difference across the invented rounds?
5. Why might two rigorous studies obtain different numeric results?
6. What details would another person need before attempting a new airplane experiment?

---

Reflection · The strength to return

A first result can open a door. The second attempt tests whether the door leads where we thought. Science grows stronger not by banning uncertainty, but by making the steps available to be examined again.

Takeaway: Record the method. Check the calculation. Gather new observations. Compare carefully. Report uncertainty. Revise what the evidence supports.

Completion: You have completed Lesson 012 when you can distinguish computational reproduction from independent replication, reproduce the worked averages, and propose one clear improvement to the test procedure.

Sources & further learning

• National Academies (2019) — Reproducibility and Replicability in Science, definitions.
• National Academies — Reproducibility chapter.
• NIST Engineering Statistics Handbook — Experiments and Experimental Design.

← Review Lesson 011 · Explore All Twelve Lessons · Visit the Quantum Atrium

Nova Academy is a free, non-accredited learning resource of the Electric Icarus Project. The airplane measurements and comparison rounds are invented teaching data, not results of a performed study.