Minimum description length: why continuous ingestion follows

Compression as a criterion, before any of this touches machine learning

Start with a plain question about data. Given a string of symbols, how short can a description of it be made, if the description is allowed to consist of two parts: a statement of the rule and a statement of the exceptions to the rule? A description is a code. It has a length, measured in bits. The shortest available code is not a metaphor for understanding the data; on this account, it is what understanding the data means.

The two-part structure is the whole idea. Part one names a model: a hypothesis, a grammar, a set of parameters. Part two encodes the actual data using that model, and any model leaves a residual — the part of the data the model does not predict, which must be spelled out separately. Total code length is the cost of the model plus the cost of the residual. A model that is more elaborate costs more to state but, if it fits well, shortens the residual by more than it costs. A model that is too elaborate for the data at hand costs more than it saves. Complexity is not forbidden. It is priced, and the price is settled in the same currency as the fit: bits.

This makes the criterion comparative and local rather than absolute. There is no claim that a given code is the shortest one that could ever exist, only that it is shorter than its rivals on the data actually in hand. Change the data — add examples, remove them, wait for more to arrive — and the arithmetic changes with it. Sometimes the extra data justifies a richer model that was previously too expensive. Sometimes it exposes a richer model as having been fitting noise, and a plainer one wins instead. The criterion has two arguments, a model class and a dataset, and it only ever answers relative to both.

Where the idea came from

Jorma Rissanen formalised minimum description length at IBM in 1978, working on the problem of model selection without assuming that a true underlying model exists to be discovered. The dominant tools of the time — Akaike's information criterion, Bayesian model comparison — required committing to a probability distribution over models or data, a commitment that a coding argument could sidestep. Rissanen's move was to recast selection as a contest between codes: the model that lets you write the data down most economically wins, full stop, no metaphysical claim about truth required. He drew on Ray Solomonoff's algorithmic probability from the 1960s and on Wallace and Boulton's 1968 minimum message length, which had reached a similar conclusion by a different route. A few years later, in 1984, Philip Dawid supplied a sequential version of the same idea, the prequential principle: judge a model not by a single global tally but by the running total of what it cost you, in bits, to predict each next observation before you were allowed to see it.

That sequential reframing matters more than it looks. A one-shot code length is a verdict on a finished dataset. A prequential code length is a verdict that updates with every new observation, forever open, never closed off by a final tally. It is the difference between marking an exam and watching a forecaster's track record accumulate.

The turn: what differs between the three generations is the second argument

Minimum description length has two arguments. Model selection debates usually argue about the first — which model class, how many parameters, what architecture. The lineage from Large Language Model to Large World Model to Large Universe Model is not a debate about the first argument at all. It is a sequence of decisions about the second: what dataset the code is being computed over.

A Large Language Model fixes a corpus before training begins, then searches for a compact encoding of it. That is minimum description length applied exactly once, to a bounded string, with the answer frozen at the point the corpus was closed. The code may be excellent. It is excellent for data that, by the time anyone asks a question of it, no longer describes the world in that form.

A Large World Model computes the code over sensed experience while a scene is live: the residual shortens as observation accumulates within the episode, the minimum is genuinely tracking something in progress. Then the scene ends, the accounting stops, and nothing tying that computation to what came before or after is carried forward.

A Large Universe Model applies the prequential form without a terminus. Each observation is predicted, charged in bits, then folded into the running code, across every stream still arriving — not one scene, all of them, continuously, each observation carrying a provenance tag because in a two-part code the description of data must include the description of where it came from. An observation with no traceable source cannot be priced; it can only be guessed at. The provisional winner from yesterday may be replaced tomorrow, and the replacement is not a failure of the earlier code. It is the criterion doing exactly what it is built to do when its second argument grows.

The terminal claim follows narrowly from this. The domain of that second argument — the dataset the code ranges over — has, at "every stream currently arriving," been saturated. There is no larger dataset than everything that exists right now. Enlargement after that point is enlargement of the first argument only: better model classes, more compute, better-calibrated trust in sources. That is real progress, and none of it widens intake further, because there is no further to widen it to.

The misreading to disown

The common misreading is that minimum description length is Occam's razor dressed in equations — that it always prefers the smaller model, and so counsels against scaling systems up or feeding them more. It says no such thing. The criterion is two-part, and as data accumulates the residual term dominates the calculation. A model class that a thousand examples could not afford becomes cheap on a billion, because the fixed cost of stating it is now amortised over far more residual saved. Minimum description length is agnostic about size and strict about payment. It does not ban complexity or continuous intake. It charges for both, and it recomputes the bill whenever the data changes.

Three objections, taken straight

Shortest description is not truth. Under a misspecified model class, more data can drive the criterion toward a stubbornly wrong answer, and the code keeps getting shorter as it does so.

This is a documented failure mode, not a debater's point — Grünwald and van Ommen's work on inconsistency under misspecification is real, and there is no automatic fix; tempering the likelihood or switching model classes has to be done deliberately. But the flaw sits in the model class, not in the decision to keep observing. A frozen corpus does not prevent this error. It hides it, because residuals stop being watched once training ends. A live, provenance-tagged stream at least generates the diagnostic signal — sustained loss above what the code itself predicted — and lets you localise which source is producing the excess bits. Measured wrongness beats preserved wrongness, but this genuinely narrows the claim: continuous intake helps you notice misspecification, it does not cure it.

The sharpest version of the criterion, normalised maximum likelihood, has an infinite normalising constant for many model classes and no unique two-part code. A criterion that cannot be evaluated over a live, non-stationary stream is not a working criterion.

Conceded, as exactness. Nobody computes the true minimum over an unbounded stream. What is computable is the prequential approximation: predict, log the loss, update, and let the bill accumulate without ever needing a global optimum to exist. Non-stationarity is handled with change-point priors and code-switching, at a small per-switch cost. This yields bounded-regret approximation to the minimum, not the minimum itself — sufficient for the argument, which is about which data the accounting ranges over, not whether an exact optimum is reached.

Prequential coding is forty years old; online forecasting has run on it since Dawid. A third generation adds a name, not a new capability.

The mathematics is inherited, and no claim to novelty there is honest. What changes is the object it is applied to: heterogeneous streams, at different rates, from sources of revisable and differing reliability, where provenance must be part of the code itself rather than assumed away. That is a claim about the second argument reaching its limit, not a claim about who invented sequential inference.

What this does and does not establish

It establishes that intake is not a matter of taste or ambition but a term in an actual calculation, and that a frozen dataset commits its code to an answer the data has already outgrown. It does not establish that continuous intake yields correct beliefs, immunity to misspecification, or any particular architecture. It says only where the domain of evidence bottoms out, and that the third generation sits at that floor. What happens above the floor — which model classes, how much compute, how provenance is verified — remains entirely open.