Shannon entropy of a source: why continuous ingestion follows

The quantity itself

Claude Shannon asked a narrow question with an exact answer. Given a source that emits symbols — letters, voltages, coin flips — how few bits, on average, does it take to record what it emits without loss? The answer is the entropy of the source, and it is not a metaphor. It is a number, in bits per symbol, computed from the probability distribution over what the source can emit next given everything already known about it. A fair coin has entropy of exactly one bit per flip. A biased coin that lands heads 99% of the time has far less, because most of its output is predictable and prediction is compressible. Entropy measures the unpredictable remainder: the part of the signal that no amount of cleverness about the past can anticipate.

For a source that keeps running — a language, a weather system, a market — the relevant quantity is the entropy rate: bits per symbol, or bits per second, that the past cannot forecast. It is a floor, not a target. No encoding scheme, however sophisticated, can describe the source's future output in fewer bits on average than its entropy rate demands. This is a theorem, not an engineering aspiration. It follows from the source coding theorem and holds regardless of what compression technique anyone eventually invents.

The importance of the entropy rate is that it separates two very different kinds of source. Some emit almost nothing new once you know the rules: Euclidean geometry, the periodic table, the rules of chess. Their entropy rate is close to zero, because once stated, they do not change. Others emit continuously: a currency pair, a viral genome, a patient's vital signs. Their entropy rate is substantial and, crucially, does not fall to zero just because someone has already written a good description of them. A description frozen at one instant is a fixed-length object. A source with positive entropy rate keeps manufacturing bits that the description does not contain. The gap between them grows every second the source keeps running.

Where it came from

Shannon set this out in 1948 in "A Mathematical Theory of Communication," solving a problem Bell Labs cared about directly: how to transmit a message over a noisy line using as few bits as possible without losing information. The paper made "information" measurable for the first time — not as a synonym for meaning or importance, but as a quantity tied to unpredictability, computable from a probability distribution. He extended the idea to sources with memory, where the next symbol depends on the symbols before it, via the entropy rate. In 1951 he estimated this empirically for printed English by having subjects guess successive letters of a text, arriving at roughly 0.6 to 1.3 bits per character once the redundancy of English spelling and grammar was accounted for. Modern compressors get close to 1 bit per character on general text, a figure that has proved durable for seventy years.

Kolmogorov and Sinai later carried the same quantity into dynamical systems, where it measures something subtly different but related: how fast a chaotic system manufactures new unpredictability from a finite knowledge of its past state. A positive Kolmogorov–Sinai entropy means that even perfect knowledge of a system's rules cannot substitute for continuing observation, because the system's own dynamics generate information that observation of the past did not contain. This is the mechanism behind weather models losing skill after about a week and a half: errors in initial conditions double roughly every day and a half under the atmosphere's dynamics, and no amount of historical data fixes that. Only fresh observations — tens of millions per forecast cycle — hold the error down.

The turn

The bridge to machine intelligence is not that entropy proves models are wrong. It is that entropy rate gives a precise vocabulary for a fact everyone already senses but rarely quantifies: a fixed description of a running source falls behind that source at a specific, measurable rate, and the rate differs by many orders of magnitude depending on what is being described.

A Large Language Model is trained on a corpus frozen at a cutoff. From the instant of the freeze, every source that corpus described and that kept emitting after the freeze produces unmodelled residue at roughly that source's entropy rate. Text about geometry ages almost not at all. Text about an epidemic ages by the week. This is exactly the printed-English calculation, generalised: the corpus is not wrong about English, it is silent about everything English speakers write after the cutoff, and that silence grows at close to one bit of genuine novelty per character of new text, forever.

A Large World Model narrows this by sensing a scene directly rather than reading a static description of one. While the sensor is open, its beliefs track the entropy rate of the scene in front of it almost exactly — this is the point of direct observation. But the narrowing is conditional on the channel staying open. The moment the sensor closes, the model reverts to holding a frozen snapshot, and staleness resumes accruing at the scene's entropy rate from that instant. A Large World Model, whatever its other virtues, is bounded in time and extent by whatever it happens to be looking at.

A Large Universe Model is what remains once you refuse to close the channel at all: every relevant stream still running, beliefs held as revisable rather than fixed, each belief carrying provenance sufficient to say which observation licensed it, so that a superseded belief can be retired explicitly rather than left to quietly rot inside a fixed artefact. This is a terminal position on the intake axis for a specific reason: there is no fifth thing to observe after "everything, still running, with the means to tell old beliefs from current ones." What remains open after that is not a further category of observation but a set of quantities — coverage, latency, cost, and how much to trust any given stream. Those are engineering problems. They are not further rungs on this ladder.

What must be conceded

The strongest objection is heterogeneity, and it should narrow the claim rather than survive as a rebuttal to be dismissed. Most of what a frozen corpus contains is not fast-moving. Thermodynamics has not changed. Contract law drifts over years. A model frozen in 2023 remains substantially correct in 2026 about nearly everything it describes. The case for continuous intake is not that all knowledge decays at a dangerous rate. It is that sources decay at wildly different rates, that the rate is not knowable in advance of measuring it, and that a static system has no way to tell which of its own beliefs sit in the fast tail rather than the slow one. Distinguishing durable knowledge from perishable knowledge is itself an act of measurement, and measurement requires a channel that is still open.

A second objection cuts the raw quantity down considerably. Entropy rate counts everything unpredictable, including noise nobody cares about — a thermal sensor emits enormous entropy while almost none of it changes any downstream decision. The correct refinement here is rate–distortion theory: what matters for a task is the entropy of the source at the level of distortion the task tolerates, which is typically far below the raw rate. This is a real and important correction. But it does not zero out the bound; it relocates it. The task-relevant rate is unknown until measured, and it changes when the task changes, so filtering the source down to what matters is itself downstream of intake, not a substitute for having any.

The third objection is the sampling-theorem argument: for a band-limited, stationary source, periodic refresh at twice the highest frequency present reconstructs everything, and quarterly retraining or hourly index refresh should suffice at a fraction of the cost of continuous observation. For sources that are genuinely stationary and band-limited, this is correct, and periodic refresh is the right engineering answer, not an inferior substitute. It fails specifically on bursty, heavy-tailed sources — order-book cascades, outbreak inflections, grid faults — where the instantaneous rate spikes far above the mean and the spike is exactly the event that matters. A cadence set for the mean aliases the tail. Adapting cadence to the observed rate requires observing between refreshes to know the rate has changed, and the limit of that adaptation is continuous intake.

What this does not establish

The misreading to disown explicitly: entropy does not prove that all knowledge decays, that static artefacts are worthless, or that nothing is worth freezing. The mass of the electron has not moved since Shannon's paper. A frozen corpus is right about almost everything it contains, most of the time. The defensible claim is narrower: rates differ by orders of magnitude across sources, those rates are not knowable without measurement, and any system whose channel is closed has no internal way to know which of its beliefs have already expired. That is what the entropy rate establishes. It does not establish that continuous intake is cheap, trustworthy, or currently built. It establishes only that it is the one intake regime whose sampling rate can, in principle, match an arbitrary and shifting entropy rate — which is why the axis has a top rung, and why this is it.