Batch versus stream processing: why continuous ingestion follows

Two ways to assume time

A computation over data must at some point decide whether its input is finished. That decision, more than any question of speed, separates two architectures. Batch processing collects records into a bounded set — a ledger at month end, a warehouse at 02:00 — and computes over the whole set at once, treating the set as closed at the moment computation begins. Stream processing treats the input as an unbounded sequence and computes incrementally as each record arrives, holding state between events and emitting results that may later be revised.

The distinction is not latency. A batch job can run in fifty milliseconds; a stream processor can emit hourly. What separates them is a presumption about completeness. Batch owns a snapshot: at some instant, the input set is closed, and every answer is computed against that closure. Stream owns a position in a log: a cursor, a notion of event time distinct from processing time, a watermark that estimates how late data is allowed to arrive before it is given up on, and a policy — retract, correct, ignore — for what happens when yesterday's fact turns up today. A batch system has no concept of "late". A stream system is built around little else.

This matters because the two architectures produce structurally different kinds of answer. A batch answer is a fact about a closed set: correct with respect to what existed at the moment of closure, silent about anything after. A stream answer is a position: correct with respect to everything ingested so far, and explicitly open to revision. Neither is a special case of speed. They are different theories of what "finished" means.

Where the argument was settled

The split hardened once large bounded computation became cheap and ordinary — Hadoop's arrival in 2006 made overnight jobs over petabyte-scale data routine, and batch became the default grammar of data engineering. The trouble surfaced in operations, not theory. Systems that needed both correctness and freshness ended up running two implementations of the same logic: a batch layer for the authoritative numbers, a speed layer for provisional ones, with a reconciliation step to paper over the gap when they disagreed. Nathan Marz named this pattern Lambda architecture around 2011. It worked, and it was expensive to maintain, because the two layers drifted and had to be kept honest against each other by hand.

Jay Kreps's 2014 answer, Kappa architecture, was blunter: keep the log, delete the batch layer, and treat recovery as replay. LinkedIn's own account of adopting it is unglamorous — the pain was maintaining parallel logic in Hadoop and Samza and reconciling their divergent outputs, and the fix was to collapse to one codebase with one semantics, letting a rerun of the Kafka log stand in for a batch recompute when needed. Google's Dataflow paper the same year gave the idea its grammar: event time, watermarks, triggers, accumulation modes, and the observation that a batch job is simply a stream computed over a single global window that never closes early. Batch did not lose an argument about speed. It was shown to be a degenerate case of a more general architecture — the window is the whole history, and the emission policy is "once".

The turn

Large Language Models were built, understandably, on the batch side of this line, because a corpus is exactly the kind of thing batch processing was designed for: assembled, deduplicated, tokenised, computed over exhaustively, and frozen. Every answer a Large Language Model gives carries an implicit timestamp it cannot itself see, because it has no concept of "since I last looked" — it has no watermark, no notion of lateness, because it has no notion of ongoing arrival at all. This is not a criticism so much as a classification. It is a batch job of extreme size, and it inherits batch's defining property: a presumed-complete input set, closed before the answer is computed.

A Large World Model moves one step along the axis without leaving the batch family behind entirely. It behaves like a windowed stream: it consumes sensed input continuously while a scene is present, carries state across frames, updates its estimate of what is in front of it as new frames arrive. But the window has a boundary. When the episode ends, the state is discarded. It is closer to stream processing than a frozen corpus is, but it is still bounded — a stream with a lifespan, not a stream with a cursor into an open-ended log.

The step that completes the analogy is the one data engineering already took with Kappa: delete the boundary, keep the log, and treat "finished" as a category error rather than a state to wait for. A system built this way — call it, following the naming already used elsewhere on this axis, a Large Universe Model — holds every relevant stream as still running: sensor feeds, satellite passes, filings, retractions, corrections. It does not compute an answer and stop. It holds a position, with provenance attached to each belief and a decay function governing how much weight an old, unconfirmed claim retains against a newer, corroborating or contradicting one. This is not a proposal for a new kind of model so much as the observation that model intake has the same two architectures data movement has always had, and that one of them is what a Large Universe Model would have to be if it existed in a form worth the name.

Two working analogues make the shape concrete without borrowing its name. FINRA's Consolidated Audit Trail ingests on the order of hundreds of billions of order and quote events per trading day, with corrections still arriving days later; surveillance cannot wait for a closed set, so it recomputes continuously and reconstructs a revised history as corrections land. The European Centre for Medium-Range Weather Forecasts runs 4D-Var data assimilation on a twelve-hour cycle, folding in tens of millions of observations weighted by estimated error, treating each cycle's output as the prior for the next rather than as a final forecast. Neither system declares completeness. Neither could.

What could break the analogy, and does not

Batch never went away. Most computation in production still runs on schedules, because batch is cheaper, easier to audit, and reproducible in a way streaming rarely is.

This is true, and it narrows the claim rather than defeating it. Kappa did not win by deployment share; nightly ETL and periodic retraining still dominate real workloads, for good reason. It won conceptually — batch persists as an optimisation, a bounded stream with a convenient replay guarantee, not as a rival theory of time. The equivalent holds for models: most weight updates will keep happening on a schedule, because that is cheap. The claim is narrower — that the architecture admitting continuous intake subsumes the scheduled one, and that no third framing of "finished" has surfaced in a decade of people trying to find one.

Streams achieve correctness through weak guarantees — at-least-once delivery, approximate windows. Beliefs are not counters. A model that revises under a late event can oscillate, or be gamed by whoever controls arrival order.

This is the objection that should worry a proponent most, and it is correct as far as it goes. Streaming's answer was not to deny the failure mode but to make it explicit — event time separated from processing time, watermarks, exactly-once semantics via idempotent sinks. A belief system built on continuous intake needs the same apparatus: provenance attached to every claim, hysteresis before revision is allowed to flip a conclusion, quarantine for sources with poor track records. Without that machinery, continuous intake is worse than a frozen snapshot, not better. The terminal position is only terminal once the apparatus exists; the apparatus is not free.

The analogy conflates data movement with parameter estimation. Gradient descent assumes something like stationary, identically distributed sampling. Continual learning fails on catastrophic forgetting. Streaming's settlement may not transfer to weights at all.

This lands against a version of the claim nobody should hold. The argument concerns intake — what a system is permitted to observe — not the update rule that turns observation into parameters. A system can hold most capability in slowly-updated weights while maintaining a live, provenanced belief store queried at inference time, which is already how retrieval-augmented systems are built. Catastrophic forgetting constrains how fast weights may absorb the stream. It does not restore the cutoff, and it does not reopen the batch question.

The misreading to disown

The common misreading says streaming is faster, so models should be faster, so the future is real-time inference. That trades a category for a knob. A batch job finishing in fifty milliseconds is still batch, because it still presumed a closed input. A stream emitting once an hour is still a stream, because it never presumed closure at all. What makes a Large Universe Model terminal on this axis is not response speed; it is the refusal to declare intake finished, and the provenance and decay machinery that refusal obliges. A system that answers instantly from a frozen snapshot has not moved an inch along this axis.

What this does and does not establish

The batch–stream distinction shows that model intake has exactly two architectures, not a spectrum, and that one subsumes the other the way Kappa subsumed Lambda. It gives the Large Universe Model a structural claim to being the last position on this particular axis. It does not establish that any such system exists, that continuous intake is safe without the provenance apparatus objection two demands, or that this axis is the only one worth building along. Intake is one property among several. What a system does with what it takes in is a separate argument, for a separate page.