Calibration and proper scoring rules: why continuous ingestion follows

# Calibration and proper scoring rules: why continuous ingestion follows

A forecaster who says "70 per cent chance of rain" is not making a claim that can be checked by looking at tomorrow. It can only be checked by looking at every day the forecaster said 70 per cent, and counting how many of those days it rained. If the answer hovers near 70, the forecaster is calibrated at that probability level. If it rains 90 per cent of the time after a 70 per cent call, the forecaster is underconfident there; if it rains 40 per cent of the time, overconfident. Calibration is not a property of any single forecast. It is a property of a track record — a relationship between a class of statements and the frequency of the events they describe, measured after the fact, across many instances.

This has a sharp consequence: calibration cannot be assessed at the moment of prediction. It requires two things to exist and be joined together — the forecast, timestamped, and the outcome, arriving later. Without the second, the first is just an assertion. A weather service that never checked its forecasts against rain gauges would have opinions about tomorrow, possibly good ones, but no calibration, because calibration is not a feature of the opinion. It is a feature of the ledger.

Proper scoring rules formalise the incentive underneath this. A scoring rule assigns a number to a probabilistic forecast once the outcome is known — the Brier score, the logarithmic score, the continuous ranked probability score for real-valued quantities. A rule is proper when the forecaster's expected score is optimised by reporting their true belief, not by hedging toward the average or bluffing toward the extreme. This matters because probability forecasts are easy to game: a forecaster who always says 50 per cent is never badly wrong and never usefully right, and a forecaster who says 99 per cent whenever they have a hunch will look brilliant until the one time they don't. Proper scoring rules are constructed so that neither strategy pays. Crucially, every proper score decomposes into two components: a calibration term, measuring whether stated probabilities match observed frequencies, and a discrimination term (also called resolution or sharpness), measuring whether the forecaster actually distinguishes cases rather than reciting the base rate. Both terms require the same raw material — forecasts matched to their resolved outcomes.

Origin: an institutional problem, not a philosophical one

Glenn Brier proposed his quadratic scoring rule in 1950, working on a mundane and expensive problem: how does a weather bureau pay or rank its forecasters without rewarding the ones who shout confidently and punishing the ones who hedge honestly? Before a proper score existed, confidence and accuracy were tangled together, and a forecaster's incentive was to sound sure rather than be sure. I.J. Good gave the logarithmic scoring rule in 1952, tightening the connection to information theory. Allan Murphy, in 1973, decomposed the Brier score into reliability, resolution and uncertainty — the mathematical move that let statisticians say precisely what "good forecasting" consists of, rather than gesturing at it. Leonard Savage's earlier work on elicitation, later generalised by Tilmann Gneiting and Adrian Raftery in 2007, characterised propriety in full generality: a scoring rule is proper exactly when honesty is the dominant strategy in expectation. The thread running through six decades of this work is institutional, not abstract. Someone has to be paid, licensed, or trusted, and the only defensible basis for that is a record of forecasts matched to what actually happened.

The turn

Here is where the concept stops being a statistics-department curiosity and starts bearing on machine intelligence. The three generations in this lineage — the Large Language Model, the Large World Model, the Large Universe Model — are usually compared on scale, or on modality, or on architecture. The comparison that actually matters, for this concept, is narrower and more mechanical: what is each system permitted to observe, and when?

A Large Language Model trains on a corpus frozen at some cutoff. It can be evaluated — held-out test sets, benchmark suites — but held-out data is a partition of the past, sliced off before training and checked afterward. It is not the future arriving. The model has no channel through which its own predictions, once uttered, come back around as confirmed or disconfirmed outcomes. Whatever calibration exists is computed by someone else, outside the system, on a ledger the model never sees and cannot act on.

A Large World Model senses a bounded scene — a room, a driving corridor, a manipulation workspace — and can check its short-horizon predictions against what the scene does next: did the object fall where predicted, did the pedestrian cross where predicted. This is real, internally computable calibration. It is also capped. The scene ends, the sensors have a horizon, and beyond that horizon there is nothing to join the prediction to.

A Large Universe Model is defined by intake that never closes: every relevant stream still running, beliefs held as revisable claims with provenance and decay rather than fixed weights. Once a system's intake stays open indefinitely, its own predictions — timestamped, retained — can be matched against outcomes as they arrive, indefinitely, inside the same system that made the predictions. That is not an added feature. It is the exact configuration a proper scoring rule needs to become a standing, internal operation rather than an external audit performed once, later, by somebody else.

The misreading, disowned

It would be easy to flatten this into "frozen models are uncalibrated, continuous ones are calibrated." That is wrong in both directions, and worth disowning explicitly. A well-fitted static model can be superbly calibrated on the population it was built for — clinical risk scores fitted and validated on a matched cohort routinely are. And a system with unbounded intake can be badly calibrated: it can drift, ignore its own errors, or be scored against a badly chosen metric that rewards the wrong thing. The claim is not about virtue or accuracy. It is about location. A system whose intake ends at a fixed point cannot perform this particular measurement on itself, because the evidence the measurement requires arrives after intake has already stopped.

Three objections, taken seriously

Calibration can be measured entirely from outside. Keep a ledger, wait, score it later. Continuous intake is a convenience for the evaluator, not a category shift in the model.

True, and this is precisely how weather verification and forecasting tournaments have worked for decades. External scoring is real scoring; nothing here disputes that. Where it fails is correction. A frozen model cannot act on a score it never receives — each fix is a discrete, expensive retraining event, decided by someone else, on someone else's schedule. Continuous intake makes scoring and updating the same operation, running at the cadence outcomes arrive. The distinction is not whether calibration can be known. It is whether the system that made the forecast is a party to knowing it.

Proper scoring rules assume something stable enough to score. A stream that never stops is exactly where the target drifts, so a running score measures a moving quantity and depends on an arbitrary window.

This genuinely narrows the claim. Windowed or discounted calibration estimates do embed a choice — how far back, how much decay — and that choice affects the number. But drift is a fact about the world, not a defect introduced by measuring continuously. A frozen model faces identical drift and has no mechanism to detect it at all; its miscalibration accumulates in silence, discoverable only by an outside audit, if one ever happens. Continuous intake at least renders drift visible, as a trend in the calibration term, which is what rolling reliability diagrams and change-point methods exist to track. Detectable and contestable beats invisible. It does not mean solved.

Calibration is a weak virtue on its own. A forecaster who always states the base rate is perfectly calibrated and useless. Sharpness is what carries decision value, and continuous intake does not guarantee sharper forecasts, only honest vague ones.

Correct, and the climatological forecast is the standard counterexample: always predict the seasonal average, and you'll be calibrated and worthless. This is exactly why proper scores are used rather than calibration in isolation — the decomposition penalises the base-rate forecaster on the resolution term, which measures whether predictions actually discriminate between cases. The point survives unchanged: both terms of the decomposition need outcomes matched to forecasts, and matching requires the outcome to arrive somewhere the system can see it. A closed corpus, however sharp its outputs sound, has nowhere for that arrival to happen.

What this does and does not establish

This establishes that calibration, as a measurement, has a structural precondition: intake that outlives the prediction. It shows why the third position on this axis is not an arbitrary stopping point but a terminus — once intake is unbounded and outcomes are continuously joined to timestamped, provenance-tracked beliefs, there is no further category of observation left to add. More history, wider coverage, better provenance, yes. A fourth kind of evidence beyond "everything, continuously," no.

It does not establish that any such system is well calibrated, or sharp, or trustworthy. It does not establish that frozen models are inferior forecasters — the Framingham risk equations were coherent and useful for years before accumulated cohort data exposed their overprediction. It does not resolve the drift problem, only makes it visible. The concept fixes where the measurement can live. It says nothing about what the measurement will find.