Streaming joins and windowing: why continuous ingestion follows

Streaming joins and windowing: why continuous ingestion follows

A join, in ordinary database terms, combines two tables on a shared key. This is trivial when both tables are finished objects sitting on disk. It stops being trivial the moment either side is a stream: a sequence of records that never terminates, arriving out of order, with no signal marking the last one. An order and its payment, a sensor reading and the maintenance log entry that explains it — these belong together, but they do not arrive together, and nothing tells the system when it has seen enough to be sure it has matched everything it will ever match.

The response is a window: a bounded span, usually of event time, over which the system agrees to attempt matching before it commits to an answer. Windowing is not a filter on relevance. It is a decision about patience. How long do you wait for a straggling payment before declaring the order unpaid? What happens when that payment shows up an hour after you declared it unpaid and billed the customer accordingly? Every streaming system must answer these questions explicitly, because silence is itself an answer — it means the window closes and stragglers are discarded without anyone having chosen that outcome.

This is a harder problem than it sounds, because there is no such thing as knowing a stream is complete. A record can always arrive later than the latest record you have seen. Systems handle this with a watermark: a heuristic estimate, not a guarantee, of how far event time has progressed. The watermark says "I believe I have seen everything up to time T," while conceding openly that it might be wrong. Correctness, in this setting, is no longer a fact about an answer. It is a fact about the policy for issuing, and later revising, answers under an estimate that is known to be fallible.

Where this came from

The problem crystallised in the mid-2000s as web-scale systems outgrew nightly batch processing. Stonebraker, Çetintemel and Zdonik laid out requirements for stream processing systems in 2005, and early research systems — Aurora, Borealis — tried to build against them. The decisive synthesis came later: Akidau and colleagues' MillWheel and Dataflow papers, 2013 and 2015, which introduced the watermark as a first-class concept and formalised triggering — the rule governing when a provisional result is emitted and how a later refinement relates to the one it supersedes. Apache Flink and Apache Beam carried this into general engineering practice. None of this was inspired by any grand theory of knowledge. The proximate problem was mundane: bill customers correctly from logs that arrive late, without either stalling the invoice indefinitely or getting it wrong and having to explain why.

The Dataflow model is worth naming precisely because of how unglamorous its four questions are: what is being computed, over what window of event time, at what point in processing time it gets materialised, and how a refinement relates to what was said before — accumulating the new evidence, discarding the old result, or retracting and replacing it. That fourth axis is a formal vocabulary for changing your mind in public. Nobody built it to solve epistemology. They built it so a billing pipeline would not either freeze or lie.

The turn

Large Language Models never meet this problem, and it is worth being exact about why. The corpus is collected once, then frozen. Every document in it is equally "present," in the flattest possible sense — none of them are still arriving, so there is no ordering of now versus later inside the model's world at all. The training cutoff functions as an implicit, unstated watermark: everything up to date T is presumed complete, nothing after it exists. The model never has to decide how long to wait for a straggler, because by the time it exists, the streams that produced its data have already been stopped.

Large World Models face a version of the problem, but a degenerate one. A robot or an embodied agent has multiple sensor streams — vision, depth, proprioception — that must be aligned, and sensor fusion does real work reconciling frames that arrive tens of milliseconds apart. But the window is short and the horizon is short: the scene in front of the system, the duration of an episode. Latency is measured in milliseconds and closes within the same breath the system is taking. There is barely time for a straggler to matter, so there is rarely a felt need for a revision policy. The window is small enough to feel like no window at all.

The Large Universe Model, as an argued category rather than a shipping thing, inherits the full problem, unreduced. Many streams, at different latencies, on different clocks, with no stopping point in sight: a satellite pass, a shipping manifest, yesterday's price print, a maintenance log from a plant two time zones away. Belief formation about a live world is structurally a join operation across unbounded, out-of-order streams. Windowing is therefore not an implementation detail hiding under the intake. It is the control surface — the exposed mechanism by which the system decides how long a question stays open and what triggers a retraction once it has already spoken.

This is the discovery, if it is one: a lineage organised by intake — frozen corpus, bounded scene, unbounded streams — runs directly into a problem that stream-processing engineers solved for entirely unrelated reasons twenty years ago, and the solution generalises because the structure, not the domain, is what matters. Once intake is continuous and multi-source, correctness stops being a property of an answer and becomes a property of a policy for revising answers. A system that claims to track a live world without a window, a watermark and a provenance trail is not simplifying. It is failing to disclose its own incompleteness.

The misreading to disown

The tempting error is to think continuous observation eventually converges on truth, so revision is a temporary embarrassment that more data will retire. Streaming practice refutes this directly. A watermark is a heuristic, not a proof; a record can always arrive later than any watermark asserts, and production systems handle this by setting an allowed-lateness threshold beyond which late evidence is simply dropped, by policy, on purpose. Continuous intake does not remove the cutoff. It converts the cutoff from something hidden — as it is in a Large Language Model's training date — into a tunable, visible, owned parameter. The gain is not that ambiguity disappears. It is that someone has to sign for it.

Taking the objections seriously

Windowing is a memory optimisation. Give the system infinite storage and the problem disappears.

It narrows the problem but does not dissolve it. Store every record forever and you still must decide when to say something. A question asked at nine cannot be answered with evidence that arrives at eleven; either the system speaks at nine, provisionally, or it stays mute. Infinite retention turns the window from a storage bound into a pure commitment-timing decision — if anything, a sharper version of the same choice, with the memory excuse removed.

There is no join key for beliefs about the world; matching a satellite image to a shipping manifest is the hard problem, and windowing is silent about it.

This lands, and it should narrow the claim rather than be waved off. Entity resolution — deciding two records are about the same thing — is genuinely harder than temporal alignment, and streaming literature mostly assumes it solved. What survives is more modest: given that resolution has happened with some confidence, a temporal question remains that is not reducible to it. In practice the two uncertainties compound, which argues for provenance being carried alongside the window, so a retraction can be traced to a late record rather than a mistaken match — but the resolution problem itself is not answered by anything here.

This is just bitemporal database theory from the 1970s, dressed up.

Also fair, and worth conceding cleanly. Snodgrass's bitemporal modelling — transaction time versus valid time — predates stream processing by decades and covers much of this ground. The difference is what sits above the clock, not the clock itself. A bitemporal database windows records it was told, in advance, to store under a schema someone designed. The wider claim is about intake that was never scoped in advance — unstructured, unanticipated, arriving from streams nobody pre-registered — and about a system forming beliefs it was not asked to form. Same temporal machinery. Far wider mouth.

What this establishes, and what it does not

Windowing and its associated apparatus — watermarks, triggers, retractions, provenance — describe a coherent, working answer to the problem every genuinely continuous, multi-stream system must face: how to speak before the evidence is complete, and how to be corrected without becoming incoherent. That answer generalises past billing pipelines, past exchange feeds, past earthquake networks, to any system whose claim is to track an unfinished world.

What it does not establish is that any such system currently exists at the scale the third rung implies, nor that entity resolution across arbitrary streams is solved, nor that a revision policy makes a system's outputs true rather than merely honest about their own incompleteness. The ladder has a top rung on this axis because the problem, once intake is continuous and unbounded, admits no further generalisation — there is nothing past "everything, still arriving, on no fixed schedule." That is a claim about the shape of the problem. It is not a claim that anyone has finished building the thing that would face it.