Nova Academy · Foundations of Inquiry · Lesson 030 · November 19, 2026
Learning objectives
By the end of this lesson, you can explain why testing a model on the same observations used to fit it can give misleading results; calculate mean absolute error (MAE); distinguish training, validation and test sets; and identify leakage, overfitting and changes in the data-generating process.
From sensitivity to generalization
Lesson 029 examined how predictions change when model inputs or assumptions change. Lesson 030 asks a different question: does a model work on genuinely new observations? A model may reproduce its training examples impressively and still perform poorly on new data. This is called poor generalization. Testing on held-out observations provides a more meaningful estimate of future performance, provided those observations represent the intended use.
Training, validation and test data
A training set is used to estimate model parameters. A validation set helps choose settings or compare candidate models. A final test set should remain untouched until those choices are finished. For a time-ordered forecast, train on earlier dates and evaluate on later dates: randomly mixing future observations into training can make a forecast look better than it is. For repeated measurements from the same person, device or location, grouping may be necessary to prevent information leakage across sets.
Worked example: calculate prediction error
Imagine a model predicts the time, in minutes, required to complete four independent practice tasks. Its held-out test predictions are 8, 11, 13 and 18 minutes. Actual times are 9, 10, 16 and 17 minutes. The absolute errors are |8−9|=1, |11−10|=1, |13−16|=3 and |18−17|=1 minute. Mean absolute error = (1+1+3+1)/4 = 1.5 minutes. If a simpler baseline always predicts 12 minutes, its absolute errors are 3, 2, 4 and 5 minutes, so its MAE = 14/4 = 3.5 minutes. On this tiny illustrative test set, the model’s MAE is lower by 2 minutes. That is not proof it will win on every future task.
Overfitting and leakage
A very flexible model can memorize quirks and noise in the training data. This is overfitting: training error becomes small while error on new examples remains large. Leakage happens when information that would not be available at prediction time enters training or preprocessing. For example, calculating a scaling factor from the entire dataset before splitting can contaminate a test. Fit transformations on training data only, then apply them to the held-out data.
Hands-on activity
Collect eight repeated measurements of a harmless quantity, such as the time needed to move a paper clip across a desk with a ruler. Record the first five as training data and the final three as test data before analyzing them. Use the mean of the first five as a constant prediction for each of the last three. Compute absolute differences and the test MAE. Compare with a second rule selected only from the first five. Do not change either rule after seeing the final three. Record the units, measurement method and possible sources of variation.
Reflection
Why would a model that memorizes all eight observations be an unfair comparison? When would a time-based split be better than a random split? What would happen if your measuring method changed between training and testing?
Knowledge check
1. Which set should be reserved for final, unbiased evaluation? 2. For predictions [4, 7, 9] and observations [5, 5, 10], calculate MAE. 3. Name one example of data leakage. 4. Does a lower test MAE on three observations prove universal superiority?
Answer key
1. The untouched test set. 2. Absolute errors are 1, 2 and 1, so MAE = 4/3 ≈ 1.33 in the original units. 3. Example: using the final test outcomes to select model settings, or fitting a preprocessing transformation using all data before splitting. 4. No; a sample of three is small and may not represent future conditions.
Limitations and responsible interpretation
MAE measures typical absolute error but does not indicate direction of bias or whether errors are acceptable for the application. A single random split can be unstable; cross-validation may help select models, but a final independent test is still valuable. Data drift, unrepresentative samples, correlations and unfair subgroup performance can undermine conclusions. The numbers in this lesson are invented educational examples, not research results.
Sources and further reading
Scikit-learn User Guide — Cross-validation: evaluating estimator performance; Scikit-learn — Common pitfalls and recommended practices; NIST/SEMATECH e-Handbook of Statistical Methods.
Previous: Lesson 029 — Sensitivity Analysis · Nova Academy index
