Conjugate updating and sufficient statistics: why continuous ingestion follows

The mechanics of a belief that does not grow

Start with a coin, or something that behaves like one. You want to know its bias. You begin with a prior belief, expressed as a Beta distribution over the unknown probability of success, described by two numbers: a count of prior successes and a count of prior failures. You observe a trial. If it succeeds, add one to the first number. If it fails, add one to the second. The posterior is again a Beta distribution. Nothing about its form changed; only its parameters moved.

This is conjugacy: a prior is conjugate to a likelihood when updating on new data returns a distribution from the same family, with the update visible entirely in a shift of parameters. The Beta is conjugate to the Binomial. The Gamma is conjugate to the Poisson. The Gaussian, remarkably, is conjugate to itself. In every one of these pairings, the entire history of observations compresses into a fixed number of registers, and that compression loses nothing relevant to the question being asked. That is the second half of the machinery: a sufficient statistic is a summary that carries everything the data say about the quantity of interest, so that the raw observations, once folded into it, can be discarded without loss.

Put the two together and you get recursion. Observe. Update the summary. Discard the observation. Repeat, indefinitely, without the summary ever growing. A thousand more coin flips still need two numbers. A million more still need two numbers. The size of the belief is fixed by the model — by the choice to describe the unknown as a Binomial rate rather than something wilder — not by how much has been seen. This is a statement about information architecture, not about optimism. It says that under the right modelling assumptions, the cost of holding a belief is decoupled from the volume of evidence that produced it.

Where this came from

Bayes' theorem itself is from 1763, a rule for updating belief on evidence with no comment on how expensive that update might be. The efficiency question is much younger. Ronald Fisher introduced the idea of a sufficient statistic in 1922, while working out how to extract everything usable from a sample without carrying the sample itself. Fisher characterised sufficiency; it fell to Georges Darmois, Bernard Koopman and E. J. G. Pitman, working independently in the mid-1930s, to prove the converse that mattered practically: fixed-dimensional sufficient statistics exist, in general, only for the exponential family of distributions. That is a sharp boundary, not a courtesy. Howard Raiffa and Robert Schlaifer coined "conjugate prior" in 1961, building the language for sequential decision-making under uncertainty in business analysis. And Rudolf Kalman, extending earlier work by Peter Swerling on orbit determination, produced in 1960 the engineering form of the same idea: a recursive filter that tracks a moving state from noisy measurements using almost no memory. Apollo's guidance computer ran exactly this, with roughly 2,048 words of erasable storage, taking star sightings and accelerometer readings across a quarter-million miles without ever retaining the measurement history. For a linear-Gaussian model, that history carried nothing further once folded into the state vector and its covariance.

The turn

Here is where the concept starts to matter for a question it was never built to answer. Consider the standard objection to any system that claims to observe continuously rather than once. A Large Language Model reads a fixed corpus and stops; its belief is the trained weights, and the corpus was consumed exactly once, with no path back to it. A Large World Model attends to a bounded scene, filtering a present frame from the last one, discarding pixels as it goes — recursion, but recursion closed over a scene, not over time. A system that instead claims to keep every stream running — sensor feeds, transaction logs, market prints, satellite passes, all of it, indefinitely — looks, on first encounter, like it must accumulate without bound. More time, more data, more storage. That looks like a category too expensive to exist.

Conjugate updating says this intuition mistakes the data for the belief. Bayes' rule applied recursively — posterior becomes prior, new observation arrives, parameters move — keeps a belief in fixed memory whatever the arrival rate. The Kalman filter did not need a bigger memory for the return leg of the mission than the outbound one. A Beta-Binomial model of a clinical trial arm does not need more storage after the ten-thousandth patient than after the tenth; adaptive designs such as I-SPY 2 update posterior response probabilities from two numbers per arm and drop or graduate arms without reopening the individual patient records. A book of motor insurance policies, following Hans Bühlmann's 1967 credibility formalism, prices risk from a Gamma shape and rate per policyholder rather than from decades of claim files; four hundred thousand policyholders need four hundred thousand pairs of numbers, not four hundred thousand histories.

The pattern generalises past any one instrument. What the third generation on the intake axis adds beyond continuous sensing is provenance: a record of which stream moved which belief, so that a belief can be revised, in a principled way, when a source is later found to be wrong. Provenance costs memory, and that cost is real. But sufficiency is what makes the underlying belief itself affordable regardless of stream volume. Unbounded observation does not, by itself, imply unbounded state. That is a closure argument about intake, and it is the reason continuous ingestion is a coherent category rather than an engineering fantasy.

Where it does not close

The first objection is the one a careful reader reaches for immediately: Pitman, Koopman and Darmois proved that fixed-dimensional sufficiency lives essentially only in the exponential family. Real streams have heavy tails, latent regime structure, mixtures of behaviours, nonlinear dynamics. Particle filters need more particles as dimension grows. Nonparametric posteriors are built to grow with the sample. Closure is a special case of statistical life, not its default condition, and it would be dishonest to claim otherwise. The narrower and defensible version survives: assumed-density filtering, moment matching, variational updates and sketch-based summaries deliver bounded state approximately, with quantifiable error, even outside exact conjugacy. Extended and unscented Kalman filters are formally wrong and still navigate aircraft. The claim that holds is that cost tracks the complexity of what is believed, not the volume of what was observed — a materially weaker claim than universal closure, and the right one to hold.

If a source is later found corrupt, two pooled counters cannot tell you how much of their value came from that source. Cheap state and revisability cannot both be free.

That is the second objection, and it is the sharpest, because it concedes something the first did not: full retraction genuinely needs per-source accounting, and a single pooled sufficient statistic destroys exactly the information retraction requires. The resolution available is partitioning — keeping the statistic per source, or per cohort, and recombining at query time, so that removing a source is subtraction of its counters rather than a full reanalysis. Reinsurance loss triangles handled per cedant work this way. But partitioning trades one cost for another: memory now scales with the number of provenance buckets kept, not with total observation count, and exact unlearning stays cheap only for statistics that combine additively. For anything else, it is hard. That is a genuine boundary, not a talking point to be waved past.

The third objection cuts differently again: sufficiency is sufficient only relative to an assumed model, and a model cannot use its own summary to notice that it has become the wrong model. Under drift, a posterior updated forever the same way becomes confidently stale — variance shrinking as more is seen, while the underlying process has already moved. Constant memory buys efficiency by buying blindness to exactly the thing it discarded. Forgetting factors, covariance inflation, sliding windows, change-point priors, held-out samples for goodness-of-fit — all of them work, and all of them cost something back, in tuning or in memory that sufficiency was meant to save. The conclusion this forces is not that the argument fails, but that a system observing without a stopping point must spend part of its budget on model criticism rather than on parameter updating. That cost is real, ongoing, and separate from the volume of the stream.

What not to conclude

The misreading worth naming directly: that sufficiency makes continuous observation free. It does not. It is cheap only for a fixed, correctly specified model of a fixed set of quantities. Every new question asked of old data requires new statistics that were not being kept, and there is no retrofitting them onto observations already discarded. Provenance, model criticism, and drift detection all draw on the very budget sufficiency was supposed to spend nothing on.

What the concept actually establishes is narrower and more useful: cost decouples from stream volume, not from stream count, model complexity, or the demand for auditability. Once a system observes every stream still running, there is no fourth kind of evidence left to reach for on this particular axis — that closure is real. What remains after it, and what the objections above correctly insist on, is scale, trust and time: more streams to partition, better provenance to keep, longer histories against which drift can be judged. That is where progress goes next. It does not go into a new category of intake, because there is not one left to invent.