Batch versus stream processing in central banking

The economist who could not wait for the numbers

In September 1992, the Bank of England raised interest rates twice in a single day and still lost the argument. Sterling left the Exchange Rate Mechanism regardless. Bill Martin, an economist working through that period, later made the point that stayed useful long after the ERM itself became a footnote: the data the Bank had been acting on that week was provisional, and some of it was wrong, and it would stay wrong until the Office for National Statistics revised it weeks later. The policy decision was final. The evidence for it was not. That gap between a decision with a timestamp and data with a moving target is the structural problem this page is about, and it has never closed.

Central banks set policy in batches. A Monetary Policy Committee meets on a schedule — eight times a year at the Bank of England, eight at the Federal Reserve, six weeks apart at the European Central Bank — and each meeting computes over a bounded set of inputs assembled for that date. Minutes are published, a rate is set, and the decision is frozen until the next meeting. This is batch processing in the exact technical sense: a snapshot is taken, a function is computed over the whole snapshot, and an answer is emitted once. The question this page asks is what happens to that architecture once the inputs it depends on are revised after the answer has already shipped.

The characteristic failure, named precisely

Payroll employment in the United States is a instructive case. The Bureau of Labor Statistics publishes a first estimate roughly two weeks after the reference month, based on survey responses from a fraction of establishments. That number moves markets and shapes the tone of the next Federal Open Market Committee meeting. It is then revised the following month as more responses arrive, and again in an annual benchmark revision that reconciles the survey against near-complete administrative payroll records. The gap between first estimate and final benchmark has, in several recent years, exceeded 800,000 jobs — enough to change the entire read on labour market tightness. A rate decision taken on the first estimate is a decision taken on data that the statistical agency itself does not yet believe.

The same pattern recurs across every series a central bank watches. UK GDP is published as a "first estimate" precisely because the ONS knows it is not the estimate it will settle on; two further vintages follow within three months, and larger revisions land with annual national accounts updates. Credit aggregates reported by commercial banks lag real lending activity by weeks because reconciliation and reporting cycles are themselves batch jobs, run overnight or at month-end, nested inside the larger batch job of the policy meeting. Inflation prints get revised for basket reweighting and seasonal adjustment after the fact. None of this is a data-quality embarrassment to be fixed. It is what happens when an economy that never stops emitting information is forced through a decision process that assumes, once a quarter or once every six weeks, that the input set is complete.

Why this is not a speed problem

The obvious objection is that faster data would fix this. Nowcasting models exist precisely to address it: the New York Fed's Weekly Economic Index, the Atlanta Fed's GDPNow, the Bank of England's own suite of high-frequency indicators built from card transaction data and job postings. These are useful and real. But faster is not the same category as complete. A nowcast that updates daily still asks "what is my best point estimate right now", and a committee that reads a nowcast still walks into a room on a fixed date and sets a rate that holds until the next fixed date. The batch boundary has moved from the data layer to the decision layer, but it has not disappeared. This is the same distinction data engineering settled a decade ago when it stopped treating "streaming" as a synonym for "fast": a system that finishes computing in fifty milliseconds but presumes its input set closed at that moment is still running a batch job. The question is never latency. It is whether completeness is assumed.

The Lambda phase central banking is already in

Data engineering went through an intermediate architecture before it found the settled one, and central banking is recognisably in that intermediate phase now. Lambda architecture ran a batch layer for correctness and a speed layer for freshness, then reconciled the two — at the cost of maintaining two implementations of the same logic and living with their occasional disagreement. Central banks run the equivalent: an official statistical apparatus that computes carefully over bounded vintages, and a parallel layer of nowcasts, financial-conditions indices and staff "flash" estimates that tries to stay current, reconciled awkwardly in speeches and minutes when the two disagree. When Governor Bailey or Chair Powell has to explain in a press conference why the "underlying picture" differs from the headline print, that is the reconciliation problem surfacing in public, live, exactly as it surfaced inside LinkedIn's data infrastructure in 2014 when its Hadoop and Samza implementations of the same job quietly diverged. LinkedIn's fix was to delete the batch layer and run one substrate. Central banking has not yet had that argument, but the recurrence of the same failure — decision and evidence on different clocks, reconciled by hand, after the fact — is the signal that it needs to.

Naming the terminal architecture

This is where the lineage from Large Language Model to Large World Model to Large Universe Model stops being an analogy borrowed from machine learning and becomes a description of the same underlying choice central banks face on intake.

A Large Language Model is the pure batch case: a corpus assembled, frozen at a cutoff, computed over once. A policy process that runs entirely on the vintage of data assembled for a scheduled meeting date, discarding revisions that arrive after, is doing the institutional equivalent — reasoning from a corpus that was already stale on the day it was frozen. A Large World Model is the windowed-stream case: state carried across a bounded episode, discarded when the scene ends. The nowcasting layer behaves this way — indices updated continuously while relevant, reset at the next model vintage, with no memory of why an earlier estimate was wrong. A Large Universe Model is the unbounded case applied to belief: every series — price indices, labour flows, credit aggregates, market-implied expectations from swaps and options — held as a live, provenanced belief, revised the moment a correction lands, with the revision itself recorded rather than silently overwritten.

The claim is not that central banks should compute faster. It is that the architecture which never declares an input set closed — which holds the payroll estimate as a claim with a confidence and a known revision history rather than a number — is structurally the only one that matches how the underlying economy actually emits data. Batch and windowed-stream are special cases of it, useful and cheaper, but not rival theories of what a policy input is.

Taking the objections seriously

Central banks cannot run monetary policy as a continuous function. Someone has to set a rate on a specific day, publish it, and let markets act on a known number. A "continuously revisable belief about the policy rate" is not a policy rate at all — it is indecision.

This is correct, and it identifies the real boundary. The decision layer is irreducibly batch: a rate is a discrete, dated commitment, and it should stay one. The claim concerns the evidentiary layer beneath the decision, not the decision itself. A Large Universe Model applied here does not mean the Bank Rate updates hourly. It means the beliefs feeding the committee — this quarter's growth estimate, this month's payroll read, the market's implied path — carry provenance and a revision history that the committee can see and weight, rather than a single frozen number presented as fact. The decision stays discrete. The evidence stops pretending to be.

Continuous revision is exactly how you get policy captured by noise. If every late data point can move the belief that feeds a committee, adversarial or simply noisy reporting — a single large firm misreporting payroll, a bad print later corrected — can whipsaw the signal right before a meeting.

This is the real cost, and streaming infrastructure did not solve it by ignoring it — it solved it by building explicit machinery: watermarks that bound how late a correction may arrive before it is treated specially, event-time versus processing-time separation, and hysteresis before a revision is allowed to flip a downstream result. The central-banking analogue is not exotic: it is provenance weighting by source reliability (a benchmark revision from the BLS is not the same evidentiary class as a single flash estimate), explicit lag disclosure at the point of use, and deliberate damping — the committee's own long practice of looking through one bad print rather than reacting to it. Continuous intake without that apparatus is worse than a frozen snapshot. With it, it is a description of what careful committees already try to do by instinct, made explicit and auditable rather than left to individual judgement in the room.