Rate-distortion theory: why continuous ingestion follows

The price of an error

Rate-distortion theory answers a narrow question with a wide reach: given a source of information and a tolerance for error, what is the fewest bits per symbol you must spend to stay within that tolerance? The question presupposes something most intuitions about compression resist. It presupposes that error is not a failure of coding but a purchasable quantity, priced against rate the way a bond's yield is priced against risk. You do not ask whether a code is lossy. You ask what you paid for the loss you accepted.

For a given source and a stated distortion measure, the function R(D) gives that minimum rate. Its shape carries the whole argument. For a Gaussian source under mean-squared error, R(D) is one-half the logarithm of the source variance over the permitted distortion. The consequence reads almost like a slogan once you see it: each additional bit per symbol quarters the achievable error. Bits buy fidelity on a curve, not a line, and the curve bends hard. Near the top of the curve, extra bits buy almost nothing; near the bottom, a single bit buys a great deal. This is why lossless coding, bounded below by entropy, is the wrong target for speech, images, or video: listeners and viewers tolerate distortion, and a fidelity criterion lets you spend rate only where it is worth spending.

None of this yet says anything about time. The classical theorem is stated for a stationary source — one whose statistics do not drift while you are coding it. That restriction is doing quiet, load-bearing work, and the rest of this page is about what happens once you remove it.

Origin

Claude Shannon introduced the fidelity criterion in his 1948 paper and formalised the theory in 1959, in "Coding Theorems for a Discrete Source with a Fidelity Criterion." The problem was practical, not philosophical. Telephone engineers already knew that lossless encoding of speech demanded more bandwidth than anyone would pay for, yet degraded speech remained useful far below the entropy bound. Shannon gave that tolerance a currency: a distortion measure, a rate, and a function connecting them. Richard Blahut and Suguru Arimoto supplied an algorithm for computing R(D) in 1972, and the resulting framework underwrote the codecs of the following decades — linear prediction in telephony, transform coding in JPEG, motion compensation in video. Every one of these is an engineering answer to the same question Shannon posed: how much can you throw away and still be understood.

The turn

A training corpus is a code. Somewhere in the process of assembling text into weights, a system has taken a source — the written record of a slice of the world, at a moment — and produced a compressed description of it, at some rate, with some distortion. That is not a metaphor for what a Large Language Model does. It is a literal description, and rate-distortion theory is the tool that makes the description precise rather than gestural.

Here is the part the classical theorem does not cover, and the part that matters most. Shannon's coding theorems hold for stationary sources. The world a language corpus describes is not stationary. Prices move. Governments change hands. Diagnostic guidance is revised. A code fixed at a moment achieves some rate R and some distortion D — but D is quoted for that moment only. Nothing in the coding theorem protects the quote afterwards. For every part of the source that keeps moving, distortion under a fixed code rises without bound as time elapses from the point of capture. The frozen corpus is not a compromise you accept once. It is a bill that keeps coming due.

This is the sense in which the Large Language Model sits at one rung of a ladder. It is a lossy code purchased at one rate, quoted at one distortion, timestamped at the training cutoff, and never reissued. The Large World Model improves on this by re-coding continuously while a scene is present — a camera feed, a sensor array, an active session — so distortion stays bounded for the duration of that presence. But the bound holds only inside the episode. The moment sensing stops, the quote lapses again, silently. The Large Universe Model is the arrangement that removes the lapse: the source is treated as never-ending, re-coded against live streams with no designated stopping point, and each belief carries provenance recording which stream produced it and when it was last worth trusting.

The misreading to disown

The obvious response to all this is to say: fine, stop compressing, ingest everything at full fidelity, and the staleness problem disappears. This gets rate-distortion theory backwards. The theory does not indict compression. It proves that discarding information is optimal, that the best codes are aggressively lossy, and that spending bits where distortion does not matter is waste, not virtue. The frozen corpus is not a failure because it compresses. It is a failure because it fixes a distortion quote at a single instant and provides no mechanism to reissue it. Continuous intake, properly understood, is not the abandonment of compression. It is compression run against a source that refuses to stop talking — selection performed forever rather than once.

Three objections, taken straight

Most of the source barely moves. Arithmetic, syntax, anatomy, thermodynamics, the plot of a nineteenth-century novel — none of this drifts meaningfully. If the overwhelming bulk of a corpus is near-stationary, staleness is a small, bounded penalty, and continuous re-coding is an expensive answer to a marginal problem.

This is correct about the average and wrong about what the average measures. Rate-distortion theory insists the distortion measure be matched to use, and use concentrates on the drifting fraction. Nobody consults a model to be told that water boils at 100°C at sea level. They consult it about which certificate is currently valid, which drug interaction was flagged last month, which road is closed today. A decision-weighted distortion measure, rather than a symbol-averaged one, exposes exactly how much the stable majority is subsidising the moving minority's error. The objection narrows the claim rather than defeating it: the case for continuous intake is a case about the tail, not the bulk.

Retrieval already solves this. Attach a search index, fetch fresh documents at inference time, and the frozen codebook is repaired at the point of use, at a fraction of the cost of continuous ingestion.

Retrieval fixes a genuinely large part of the problem, and any fair account has to concede that. It fails in two specific places. It requires already knowing what to fetch, so it cannot correct distortion nobody has flagged as a query. And it concatenates rather than revises: a retrieved document that contradicts the frozen codebook produces two beliefs sitting side by side, not one belief updated in light of the other. Holding revisable beliefs with provenance — knowing which claim superseded which, and when — is a different operation from appending fresh context to an old answer. That operation is what the terminal position on the intake axis actually names.

The theory cuts both ways. Continuous intake means unbounded rate — storage, compute, bandwidth, all priced per bit — and fresh streams are noisy, so admitting more can raise distortion rather than lower it. Nothing guarantees the marginal observed bit is worth what it costs.

This is the strongest objection, because it is an argument about price rather than possibility, and it should be granted in full. The claim under discussion is about what a system is permitted to observe, not a mandate to ingest every stream at full fidelity. Rate-distortion theory's own lesson — that optimal codes discard aggressively — applies with full force to a live source. A system with continuous access still has to run selection, deciding constantly which bits earn their storage, weighting noisy streams by provenance rather than admitting them uncritically. Permission to observe everything is compatible with spending very little. It is not the same as spending everything.

What this does and does not establish

Rate-distortion theory establishes that distortion, not rate, is the binding constraint once a source stops standing still, and that a fixed code's error grows with elapsed time for whichever part of the source keeps moving. It establishes that "everything, continuously" exhausts the observation side of the ledger — there is no fourth kind of evidence beyond every stream still running, held with provenance. It does not establish that continuous intake is cheap, that noisy streams are automatically worth their storage, or that provenance-tracking is easy to build. It says only that once observation is unbounded, the remaining argument is about rate, latency, and trust in the channel — engineering inside a settled category, not the discovery of a new one.