Lesson 026 — Calibration and Model Validation: Does the Model Match the World?
In Lesson 025, we learned that a model is a selective representation. Today we test whether its predictions are useful. A model can be elegant and still wrong. Validation compares predictions with observations not used to fit the model; calibration adjusts a model or instrument using known reference data. These are related but distinct activities.
Learning objectives
- Distinguish calibration, validation, training data, and test data.
- Calculate prediction errors and mean absolute error (MAE).
- Explain why testing on new observations matters.
- Identify a model’s limits and revise it responsibly.
Worked example: predicting water use
A simple model predicts liters used in four independent trials: 10, 12, 14, 16. Actual measured values are 11, 11, 15, 19 liters. Absolute errors are |10−11|=1, |12−11|=1, |14−15|=1, |16−19|=3. MAE=(1+1+1+3)/4=1.5 liters. The average signed error is (−1+1−1−3)/4=−1 liter, showing underprediction on average. A low average signed error can conceal large opposing mistakes, which is why MAE helps.
Calibration versus validation
Suppose a scale reads 102 g for a certified 100 g mass. Adjusting the scale against that reference is calibration. Measuring a different certified 200 g mass afterward and checking its reading is a simple validation check. In statistical modeling, tune parameters on training data and assess generalization on separate test data. Do not repeatedly tune to the final test set: that leaks information into the evaluation.
Activity: challenge a prediction
- Choose a measurable quantity, such as daily water consumption or travel duration.
- Write a simple prediction rule using earlier observations; state assumptions.
- Before gathering four new observations, record four predictions.
- Compute absolute errors and MAE for those new observations.
- Compare with a simple baseline such as always predicting the training-data mean.
- Explain where and why the model failed, without rewriting the original predictions.
Reflection
What would make you revise a model you find intellectually appealing? When does a prediction become an explanation, and what additional evidence would you need?
Knowledge check
- What is the difference between calibration and validation?
- For predictions 4, 7, 9 and observations 5, 5, 10, calculate MAE.
- Why should the final test set be separate from the data used to tune a model?
- Does a good fit prove that the model’s mechanism is causally correct?
Answer key
- Calibration adjusts against reference information; validation checks performance against independent evidence or criteria.
- Absolute errors 1, 2, 1; MAE=4/3≈1.33 units.
- To obtain a less biased estimate of performance on unseen cases and reduce overfitting.
- No. Different mechanisms can generate similar patterns; causal claims need additional design and evidence.
Limitations and responsible use
Small samples produce unstable error estimates. Distribution shift, inaccurate measurements, selection bias, and unmodeled conditions can degrade predictions. A model validated in one population, season, or setting may fail elsewhere. These numbers are invented for teaching, not measured research findings.
