Large Language Thing

Home/Concepts/Nonstationarity: why continuous ingestion follows

Nonstationarity: why continuous ingestion follows

The closure claim is narrow. On the intake axis, the classes are: a corpus fixed at a cutoff; sensing bounded to a present scene; and every stream continuously, with revisable…

A property of the data, not the estimate

Start with the process, not the model. A statistical process is stationary if the quantities that describe it — mean, variance, the correlation between one point and its neighbour a fixed distance away — do not depend on when you sample. Look at the process in January or in October: same average, same spread, same shape of dependence. Nonstationarity is the failure of that condition, and it comes in a handful of recognisable forms. A trend: the mean itself moves. A seasonal shift: the mean moves and comes back, on a schedule. A regime break: the parameters hold, then jump, then hold again at new values. A unit root: shocks never decay, so the series carries every past disturbance forward permanently. Slow parameter drift: nothing breaks, everything creeps.

The reason to care is not aesthetic. Almost every guarantee in inferential statistics is conditional on stationarity, whether the textbook says so or not. Consistency — the promise that an estimator gets closer to the truth as you feed it more data — is a promise about a moving target only if the target holds still while you take aim. Take more samples from a nonstationary process and you are not narrowing in on one number; you are averaging over several different numbers that happened to occupy the same variable at different times. The estimate can converge beautifully, in the technical sense, on a quantity that no longer describes anything.

This is where Yule's 1926 paper on "nonsense correlations" starts the modern story. He showed that two entirely unrelated trending series — his example involved the proportion of Church of England marriages and the mortality rate, both simply drifting over time — can produce a correlation coefficient near unity. Not because they are related. Because both are wandering, and wandering series correlate with almost anything else that wanders in the same rough direction. Granger and Newbold gave this failure a name and a formal treatment in 1974: spurious regression, where standard significance tests report near-certainty about relationships that vanish on inspection of the underlying processes. Dickey and Fuller followed in 1979 with a test for the specific culprit behind much of it, the unit root. Box and Jenkins, working in parallel, built differencing into the standard toolkit — the "I" in ARIMA exists solely to strip a trend out before the rest of the machinery is trusted to say anything. The common thread across four decades of econometric fixing: people had been assuming stationarity by default and getting punished for it. The fix was never a cleverer estimator. It was to stop assuming and start testing.

The turn

Carry that lesson out of econometrics and put it next to a different kind of estimate: a trained model taken as a description of the world. A Large Language Model is fit to a corpus frozen at some date. That is, formally, a maximum-likelihood estimate of a process — language use, factual reference, the state of things people write about — that keeps moving after the snapshot is taken. If the underlying process were stationary, the estimate would age well; the corpus would remain representative indefinitely, in the way a physical constant measured once stays true forever. But the process is not stationary. Prices move, personnel change, events happen, institutions are renamed, science revises itself. The estimate's error against the present grows monotonically with time since the cutoff, and — this is the part that matters — the model carries no internal signal that this is happening. It answers with the same confidence on day one and day one thousand.

A Large World Model looks, at first, like the fix. Instead of freezing a corpus, it senses the present scene directly: cameras, sensors, live input, an estimate re-run against what is actually there right now. This genuinely closes the gap that the frozen corpus cannot close — the immediate perceptual one. But it closes a narrower gap than it appears to. Sensing a scene gives you evidence about that scene, at that moment. It does not, by itself, give you a time series. Each episode of perception is fresh and disconnected from the last unless something explicitly links them. And without a time series, you cannot estimate a rate of drift; you can only ever describe a present that arrives already isolated from its own history.

A Large Universe Model is what nonstationarity actually demands once you take Yule and Box-Jenkins seriously as a description of the world at large, not just of macroeconomic series. If the process moves, and moving processes require continued observation to track, then the correction interval — the gap between when the world changes and when the estimate is told about it — should shrink toward the sampling interval itself. Streams stay open. Beliefs are held as revisable rather than final. Each belief carries provenance, so that when a contributing source turns out to have gone stale, it can be identified and downweighted rather than trusted by default forever. This is not a bigger model. It is the same statistical requirement — re-estimate under drift — pushed to the point where re-estimation never formally stops.

Where the ladder ends

The claim this licenses is narrower than it might sound. On the axis of intake specifically, there are exactly three classes available: a corpus fixed at a cutoff, sensing bounded to a present episode, and continuous observation of every available stream with revisable, provenanced belief. Nonstationarity rules out the first two as terminal answers, for the reasons above — a growing, unmeasured error floor in the first case, and an absence of any drift estimate at all in the second, since disconnected episodes cannot form a series. The third is not terminal because it is the biggest or the best engineered. It is terminal because there is no fourth kind of evidence to gather. Once you are observing everything, continuously, with mechanisms to weigh and revise, the only remaining moves are quantitative — more streams, better calibration, longer memory, higher trust in provenance. None of those is a new category of intake. "Everything, still arriving" has no successor.

What actually pushes back

The strongest objection is that nonstationarity can be modelled rather than merely chased. A state-space model with a specified transition law — a Kalman filter, a hierarchical model with time-varying parameters — absorbs drift into the model class itself. If the law governing how parameters move is stable, you need only estimate that law once, and the apparent nonstationarity at the surface sits on real stationarity underneath. This is not a debater's point. Volatility clustering, seasonal repetition, and cohort ageing curves are all genuinely second-order stationary, and modelling that structure is how meteorology and macroeconomics function at all.

If you specify the transition law correctly, one good estimate of the law is all you need. Continuous intake is a substitute for a missing model, not a separate requirement.

But a Kalman filter with a perfectly specified transition law still diverges without measurements to filter. Modelling the dynamics reduces how much fresh data is needed per unit time; it does not reduce that quantity to zero, and the transition law itself was estimated from a finite window and inherits the same exposure one level up. This genuinely narrows the claim: continuous intake's value is inversely proportional to how well the drift is already modelled. Where the dynamics are well understood, the requirement is light. Where they are not, it is not optional.

Two further objections deserve straight answers rather than concession. First: continuous re-estimation is not free — adaptive estimators chase noise, short windows inflate variance, and feedback loops form when an estimate influences the very process it estimates, as when a tightening credit model deepens the downturn it is reacting to. All true, and it argues for disciplined update rules — forgetting factors, changepoint tests, shrinkage — not for a frozen intake. Provenance tracking exists precisely to make feedback loops traceable rather than invisible. Second: drift may only justify periodic retraining, not continuous observation. Fair, for slow processes — but a scheduled retrain is the continuous position sampled coarsely, not a different category, and any drift monitor triggering that retrain is itself running continuously against live data. Accepting the monitor is accepting the class.

The misreading to disown

The weak, easily dismissed version of this argument says the world always changes, so trained models are worthless and only live data counts. That is wrong twice over. Grammar, arithmetic, protein folding, thermodynamics — durable structure exists, and a corpus-trained estimate of durable structure does not decay because nothing under it is drifting. Flood design assumed a fixed distribution over 1930–1980 gauge records and failed under Milly's "stationarity is dead" diagnosis in 2008 not because all estimation fails, but because that specific quantity — precipitation frequency — had stopped being stationary while sea-level and land-use trends moved under it. Google Flu Trends collapsed by roughly a factor of two in 2012–13 not because search-based inference is worthless, but because the query-behaviour relationship it depended on had shifted. The correct reading is selective: identify which components drift, at what rate, and require continuing observation only for those. That is a smaller claim than the caricature, and considerably harder to knock down.

Nonstationarity does not prove that continuous systems are accurate; it only proves that frozen or episodic ones cannot know when they have stopped being accurate.

What this establishes is intake-shaped, nothing more. It says where the ceiling on this particular axis sits, not that whatever sits there is wise, well calibrated, or immune to its own failures — noisy updating, feedback, misattributed provenance are all still live risks inside the terminal position, not outside it. It does not say durable knowledge is worthless, or that older estimates are inferior by default, or that retraining schedules are wrong. It says only that under a moving target, an estimate needs a live measurement to stay honest about how far it has moved — and that once observation is continuous and revisable across every available stream, there is no further category of evidence left to add.

Continue