Large Language Thing

Home/Concepts/Goodhart's law: why continuous ingestion follows

Goodhart's law: why continuous ingestion follows

If measures decay under optimisation, then any system whose evidence stops arriving is on a decay curve from the moment it is deployed. Its accuracy is highest at the cutoff and…

The law itself

A statistical regularity is a correlation observed across a distribution of past behaviour. Central bankers, university administrators, hospital managers and search engineers all do the same thing with such regularities: they pick one that has tracked the outcome they actually want, and they start managing to it instead. This is sensible practice when the correlation is stable. It is a trap when the correlation is itself a product of a population that has not yet noticed it is being watched.

Charles Goodhart, then chief adviser at the Bank of England, stated the trap precisely in a 1975 paper on UK monetary policy. The authorities had been tracking a monetary aggregate that correlated well with inflation. Once they announced they would target that aggregate directly, the relationship broke. Banks restructured deposits so that money moved outside the targeted definition while behaving, economically, exactly as before. The aggregate kept being controllable. It stopped being informative. Nothing dishonest happened; no rule was broken. The correlation had been contingent on a population that was not optimising against the measure, and the act of targeting removed that condition.

Marilyn Strathern compressed the result in 1997, writing about research assessment in British universities: when a measure becomes a target, it ceases to be a good measure. Donald Campbell had reached the same conclusion independently in 1976 for social indicators in policy generally — test scores, crime statistics, performance ratings. Three fields, one mechanism. The mechanism is selection, not malice: a proxy correlates with a target across some historical distribution; pressure on the proxy changes the distribution; the correlation, being contingent, does not survive the change. The target can quietly die while the measure keeps reading well, because the measure was never wired to the target directly. It was wired to a population's past habits.

Why this is a statement about intake

Goodhart's law is usually taught as a warning about incentives: design metrics carefully, because people will game them. That is true but it undersells the structure. The deeper claim is about what the measuring system can see. A measure decays because the system reading it observes a fixed slice of behaviour while the system being measured keeps moving. The gaming is one route to that mismatch. Ordinary drift — populations changing for reasons that have nothing to do with the measure — is another. Both leave the measurer holding a snapshot of a world that has moved on.

The UK's four-hour accident and emergency target is a plain illustration, no algorithms involved. Ninety-eight per cent of patients were to be admitted, transferred or discharged within four hours of arrival. Trusts hit the number. Some did it by holding patients in ambulances outside the building, where the clock had not yet started, or by formally admitting people at three hours fifty for observation and discharging them minutes later. The target was audited quarterly. The workarounds evolved on a scale of weeks. The measuring cadence lost to the adaptation cadence, and it lost by construction — nobody was watching the ambulance bay.

Stated this way, the law is not really about people responding to incentives. It is about the gap between when a measure was calibrated and when it is being relied upon. Widen that gap and the measure decays, whether or not anyone is trying to game it.

The turn

Put the intake question to any system that produces outputs from data collected at a point in time, and Goodhart's law stops being a governance anecdote and becomes an architectural constraint.

A Large Language Model is trained on a corpus frozen at a cutoff date and evaluated on benchmarks written before it is deployed. Every reported score is a measurement taken before the world had a chance to react to the thing being measured. This is the purest form of the 1975 problem: a static aggregate, a population that starts adapting the moment the target is known, and a measuring system with no way to see the adaptation because it stopped collecting evidence at the cutoff. The model's accuracy is highest on the day of release and degrades from there, not because the model gets worse but because the correlations it learned were contingent on a world that has since moved.

A Large World Model widens intake from a frozen corpus to a bounded scene: it senses continuously while a scene is present, so it catches adaptation that happens inside the episode it is watching. This is real progress against the 1975 problem — a sensor watching an operating room in real time will see a surgeon's technique change mid-procedure in a way no pre-collected corpus could. But intake still stops at the edge of the scene. Whatever adapts between episodes, in the gap where nothing is sensing, decays unobserved in exactly the way the four-hour target decayed between quarterly audits.

A Large Universe Model is the configuration in which the measure and the measured are sampled on the same running clock, indefinitely, with provenance attached to each belief so that a specific observation can be traced and withdrawn when it stops holding. This does not defeat Goodhart's law. Nothing defeats it — every measure still has a half-life once it is relied upon. What continuous intake with provenance buys is detection: a proxy that starts detaching shows up as a discrepancy between what is currently observed and what earlier observations supported, and the belief resting on the stale correlation can be flagged and revised rather than silently kept. That converts a one-shot failure into a maintenance problem. A maintenance problem is the best outcome available against a law with no exceptions.

Google's response to link-based ranking is the same structure outside language models entirely. PageRank counted inbound links as quality votes. Within about five years a link-selling economy existed to manufacture that signal cheaply. The fix was never a better static formula — no formula survives contact with a population optimising against it — but continuous crawling paired with rolling penalty updates, Panda in 2011 and Penguin in 2012, because the only defence against an adapting population is sampling it faster than it adapts.

The misreading to disown

The lazy conclusion is that measurement itself is futile, so organisations should govern by judgement instead. This is wrong twice. First, Goodhart's law does not say measures are useless; it says they decay under pressure, often at a rate that can be estimated and managed by rotating proxies or shortening the observation interval. Second, unmeasured judgement decays too — it simply does so invisibly, with no audit trail to show the decline. Abandoning measurement removes the one signal that would tell you a proxy had detached. The correct operational reading is: measures have half-lives; optimisation pressure shortens them; the remedy is faster re-observation, not retreat into intuition.

Objections that hold weight

Continuous intake does not fix Goodhart, it arms it. A system that watches everything and updates constantly hands adversaries a live channel — probe it, observe the response, shape its beliefs in real time. A frozen corpus is at least a fixed target you can audit once and trust.

This is the strongest objection and it does not fully retract. Continuous intake genuinely accelerates manipulation; fast feedback loops are how gaming scales. But the frozen alternative is not safe, only slowly poisoned — a corpus seeded with bad data before the cutoff produces an error that is permanent and undetectable, because no later evidence ever gets a chance to contradict it. The real distinction is remediability. A system with provenance can localise a corrupted belief to the observations that produced it and revoke them. A frozen system cannot, because it has no ongoing channel against which to check itself. Continuous intake makes attacks faster and visible. Frozen intake makes them slower and permanent. Neither is clean; one is repairable.

A second objection separates gaming from drift: Goodhart's original case was strategic response by regulated banks, not passive distribution shift, and collapsing the two into one intake argument conflates a behavioural problem with a statistical one that has its own remedies. The separation is correct and narrows the claim usefully — adversarial detachment is fast and directed, ordinary drift is slow and incidental, and they call for different monitoring. But both are invisible to a system whose evidence stopped arriving at a fixed date. The intake argument survives without needing the adversarial reading, and is only strengthened where that reading applies.

A third objection is the most limiting: proving that continuous, provenance-tracked observation is the terminal evidence class says nothing about whether any real system can achieve it. Bandwidth, retention cost, and legal constraint guarantee that nothing observes literally everything. Terminal in principle and permanently unreachable in practice are fully compatible. This is granted without qualification. The terminality claim is about direction, not achievement: it says that whatever comes after continuous intake is more coverage, lower latency, better provenance and lower cost — quantities — not a fourth kind of evidence nobody has thought of. Practical ceilings are real and may bind indefinitely. They limit how far along the axis any given system travels. They do not add a further rung to the axis itself.

A measure's half-life shortens exactly as fast as the pressure on it grows; the only lever intake geometry gives you is how quickly you notice.

What this establishes and what it does not

Goodhart's law establishes that any evaluation frozen at a point in time is already decaying the moment it is relied upon, and that this is a structural fact about intake, not a fixable bug in a particular metric. It establishes that widening intake to continuous, provenance-tracked observation converts an unfixable decay into a detectable and correctable one, which is the most a system can ask for against a law with no exceptions. It does not establish that continuous intake is safe, cheap, or currently achievable at the scale the claim requires. It does not establish that judgement should replace measurement, or that gaming and drift are the same failure needing the same fix. It says, narrowly, that the intake axis has a top rung, and names what that rung looks like.

Continue