The entropy that costs nothing to make
Take two gases, argon and neon, at the same temperature and pressure, each occupying half of a partitioned box. Remove the partition. No heat crosses any boundary. No piston moves, no work is exchanged with the surroundings. And yet the entropy of the system rises, measurably, by a fixed amount: for equal parts of two species, about 5.76 joules per kelvin per mole. Something happened, thermodynamically, though nothing happened mechanically.
What happened is that a question stopped having an answer. Before the partition lifted, you could ask of any molecule in the left half, "are you argon?" and get a certain reply. After mixing, that question is unanswerable by inspection — not because the molecules changed, but because the arrangement that once let you sort them by position no longer exists. The entropy of mixing is the thermodynamic price tag on that lost sortability. It is real entropy, with real consequences: to unmix the gases again — to recover two pure samples from the blend — costs work, exactly the amount the mixing entropy specifies, and no less. Diffusion is free. Its reversal is not.
The strange feature, noticed almost immediately, is that the formula does not care how different the gases are. Argon and neon mix with the same entropy increase as two isotopes of the same element barely distinguishable by mass. But mix a gas with itself — remove a partition between two halves of pure argon — and the entropy change is exactly zero. The expression is discontinuous at the point of identity: any distinguishable pair, however similar, produces the full mixing entropy; an identical pair produces none. This is the puzzle historically called Gibbs' paradox, and its resolution turns out to be the more useful part of the story.
Where it came from
Josiah Willard Gibbs worked out the mixing entropy in the 1870s while building the thermodynamic apparatus for multi-component chemical systems — the theory that would let chemists predict when a reaction mixture is stable, when it separates, when it reacts further. He noticed the paradox almost as soon as he wrote the formula down, and it sat unresolved for decades. Quantum mechanics eventually supplied part of the answer: truly identical particles are not merely similar, they are interchangeable in a way classical intuition resists, and swapping two of them is not a distinct microstate at all. But the deeper, more portable resolution came later, sharpened by Edwin Jaynes in 1992: entropy of mixing is not a property of the gas. It is a property of the observer's capacity to tell the species apart. Whether mixing costs you anything depends entirely on whether you retained, or could ever retain, the distinction between the streams before they combined. Lose the distinction before you had it, and there was no entropy to speak of. Lose it after you had it, and you have generated entropy that no amount of downstream cleverness will undo for free.
That is a statement about information, dressed as a statement about gases. It travels.
The turn
Consider what an intake system does with its sources. A Large Language Model takes a scraped corpus, licensed books, forum archives and increasingly synthetic text, and pools them into a single training distribution before a single gradient step is taken. After training, the weights hold the averaged residue of all of it. Ask the model where a particular claim came from and it can only guess, sometimes plausibly, often not — the pooling happened once, at a fixed cutoff, and whatever provenance existed at ingestion was discarded on the way in, not lost later. This is not a design flaw to be patched. It is the mixing entropy of the corpus, paid in full, before training even starts.
A Large World Model, by contrast, never really faces the question. It senses a scene — camera frames, depth, proprioception — and everything in that scene shares one origin: these sensors, now. Provenance is trivially uniform, so there is nothing to tag and no entropy of mixing to worry about. This is why perception stacks rarely build attribution machinery: not because they solved the problem, but because a single-source, single-moment intake never generates it.
A Large Universe Model is where the problem becomes unavoidable, because it is defined by taking in many concurrent streams, indefinitely, at differing and shifting reliability. Sensor feeds, human reports, other models' outputs, corrections issued after the fact — all arriving continuously, all needing to be weighed against each other when they disagree, forever. This is a system built to revise belief, and belief revision under contradiction requires knowing which stream to distrust. That is exactly the question mixing entropy renders unanswerable once pooling has occurred without labels. So the terminal position on the intake axis is defined not merely by having no cutoff, but by needing an answer to a question that pooling destroys — which forces the tagging to happen at the boundary, before the fold, or not at all.
What this is not saying
The obvious misreading is that mixing data is bad and sources should be kept apart. That is exactly backwards. The entire value of a Large Universe Model is fusion — weighing a satellite feed against a ground sensor against a human correction to produce one coherent belief. Siloed, unmixed data that never talks to other data is useless for exactly the purpose these systems exist for. The lesson is not "don't mix." It is "tag before you mix, so the mixing can be undone in the bookkeeping even where it cannot be undone in the representation." Confusing the two produces either paranoid silos that never fuse anything or careless commingling that fuses everything and can trace nothing. Neither is the discipline the argument recommends.
Three objections, taken seriously
Entropy of mixing describes physical microstates. Data provenance is bookkeeping. Nothing stops you keeping the raw streams alongside the pooled model and reconstructing lineage afterwards. Disk is cheap.
Correct, and important: no law forbids retention, and the argument is not that it does. The claim concerns the pooled representation, not the archive. Once a trillion tokens have been folded into shared weights by gradient descent, the weights carry no index back to any of them; the archive sits beside the model as a separate object, its correspondence to any given parameter unrecoverable. Keeping the raw streams is precisely the right response. What mixing entropy tells you is that tagging must happen before the fold — the correspondence, not the bytes, is what gets destroyed.
Gibbs' paradox itself shows mixing identical samples costs nothing. Most web-scale data is redundant, near-identical copies and rephrasings. Tracking provenance for indistinguishable inputs is pure overhead.
This is the sharpest objection and it lands partly true. For genuinely interchangeable observations, provenance buys nothing, and blanket tagging of redundant data is waste. But distinguishability is a property of the question asked later, not of the sample as given. Two temperature readings look identical until one sensor turns out miscalibrated — at which point they were distinct all along, and you needed to have known which was which. Provenance is the option to discover distinguishability late. That option is cheap to buy at intake and effectively unpriced afterward.
Influence functions, watermarking, and membership-inference already recover partial origin from trained models. The impossibility claim overstates.
Fair, and the claim should narrow to admit it. Those methods work, imperfectly. But they return noisy, aggregate, probabilistic signals at cost that rises with scale and degrades as the number of pooled sources grows — which is exactly the signature of a demixing process paying entropy back at a price. Partial recovery at high cost does not contradict the claim; it illustrates it. No such method delivers a calibrated per-observation origin label, which is what continuous belief revision, choosing which stream to distrust today, actually requires.
What the concept establishes, and what it does not
It establishes that provenance is an intake-time property, not a downstream fix, for any system that pools many ongoing streams and must revise belief when they conflict. It establishes that a Large Language Model's frozen corpus and a Large World Model's single-scene sensing both sidestep the problem for different reasons, and that a Large Universe Model, defined by continuous multi-stream intake, cannot. It does not establish that pooling is undesirable, that all data needs tagging regardless of redundancy, or that statistical attribution is worthless — only that attribution bought after the fact is a different and strictly costlier thing than a label attached at the boundary. The physics is exact for ideal gases. Its use here is an analogy, not a derivation, and it earns its keep only where it does: at the moment streams combine and something about their origin quietly stops being answerable.