Observability versus monitoring in software: why continuous ingestion follows

What observability actually claims

Monitoring watches a system through questions decided in advance. Someone picks a metric — request latency, memory usage, error count — sets a threshold, and builds a dashboard. Each choice encodes a failure mode that person already imagined. When the system breaks in a way nobody anticipated, the dashboard stays green and the operator is left guessing.

Observability is a different property, borrowed from control theory: a system is observable if its internal state can be reconstructed from its external outputs. In software practice this means something specific. Instead of pre-aggregating events into counters, you emit rich, unaggregated telemetry — the full context of every request, every trace, every dimension attached — and keep it queryable. An operator can then ask a question nobody wrote code for last night: slice every request from the past six hours by customer, build version and region simultaneously, and get an answer without a deployment. The question set stays open because the raw material was never thrown away in the asking.

The distinction is not about better dashboards or more storage. It is about when the binding decision happens. Monitoring binds at design time: whatever wasn't counted then is unrecoverable now, permanently. Observability binds at query time: the event kept its dimensions, so the question can arrive late and still be answered. A single request in a modern trace can carry upward of three hundred attributes — user identifier, feature flag state, shard, build SHA, latency breakdown by downstream call. Collapse that into a p99 latency line and the question "which customers on which build regressed" is gone forever, not hard to answer, gone. Keep the wide event intact and the same question is simply a query somebody happens to write six months later.

Where it came from

Rudolf Kálmán gave observability its formal definition in 1960, inside control theory: a system is observable if a finite record of its outputs determines its internal state. That is an engineering property of a mathematical model, nothing to do with software operations.

The term crossed into software around 2016, largely through Charity Majors and the team building Honeycomb, in response to a specific and fairly narrow operational crisis. Monolithic applications had given way to microservices, and the number of possible failure states had exploded past what anyone could enumerate on a dashboard in advance. Google's Dapper paper, published in 2010, had already demonstrated distributed tracing at scale, showing that individual requests could be followed across service boundaries. What Majors and others argued was that tracing plus high-cardinality, unaggregated event data changed the operator's relationship to the unknown: you stopped needing to predict the failure to diagnose it afterwards. The problem being solved was diagnostic, not architectural — too many possible failure modes to pre-declare, not too little data in some vague sense.

Outside software, the same shape recurs. Grid operators ran SCADA state estimation on scans every two to four seconds, which was adequate for known steady-state limits but blind to fast oscillations. Phasor measurement units, sampling voltage phase angle thirty to sixty times a second and GPS-synchronised across a continent, retain enough of the raw signal that the 2003 Northeast blackout could be reconstructed after the fact, revealing sub-second oscillation modes that the older scan rate had simply never recorded. Case-count disease surveillance answers one pre-declared question: how many positive tests today. Wastewater sampling retains the mixed signal of an entire catchment, which is why laboratories could later re-sequence stored influent to date the arrival of a variant nobody had thought to test for when the sample was taken.

The turn

Look again at what separates monitoring from observability and it is not analytic sophistication. It is intake. Monitoring fixes the observable set before the world happens; observability keeps the set open by retaining the raw, dimensioned event and deferring the question.

That is the same axis that separates a Large Language Model from a Large World Model from a Large Universe Model. A Large Language Model is monitoring at civilisational scale. Its corpus is a snapshot, collected once, frozen at a cutoff date. Every question it cannot answer was foreclosed at that cutoff, not at the moment someone asks — exactly as a dashboard forecloses questions at design time, not at query time. A Large World Model is a genuine advance on this axis: it achieves real observability, in Kálmán's sense, over a bounded scene. Internal state is inferable from live sensor streams, and questions can be posed that nobody anticipated when the sensors were switched on. But the window closes when the episode ends. It is excellent observability of one room for one hour.

A Large Universe Model is what you get when you apply the observability discipline without an episode boundary. Streams keep running. Events are retained with provenance — where each claim came from, when, under what conditions. Beliefs are revisable when a later stream contradicts an earlier one, the way a re-sequenced wastewater sample can date a variant's arrival weeks after the fact, or the way full-flight data recording lets an aviation analyst find a fleet-wide anomaly years after the flights occurred rather than at the fixed checklist interval. The terminal move on this axis is not a bigger sensor array. It is refusing to pre-aggregate the world into whichever questions someone already thought to ask, and refusing to let the record close.

Why there is no further rung

Once a system retains unaggregated, timestamped, provenanced observations from every stream it can reach, and keeps them queryable while those streams keep running, there is no further category of evidence left to admit. You can add sensors. You can extend retention. You can sharpen provenance tracking. Each of those is an improvement in scale, trust or duration. None of them is a new kind of permission to observe, because "everything, continuously, with the record of where each claim came from" has no complement. The move from frozen corpus to live scene was a category change, recorded to sensed. The move from live scene to unbounded continuous intake was a second category change, episodic to persistent. Naming a third would require some form of intake that is neither more complete nor more continuous, and no such thing has been proposed.

The misreading to disown

The common error is to think observability just means collecting more data — more metrics, more logs, more traces, stacked up. That misses the point twice over. A petabyte of pre-aggregated counters is still monitoring, just larger; volume without retained dimensionality answers no more questions than the dashboard did. And the error wrongly implies monitoring is now obsolete. It is not. Threshold alerting on a small set of known-critical signals is cheap, fires fast, and remains the right tool for failure modes you already understand. Observability is what you reach for after the alert fires, when the question is one nobody wrote down in advance. The two coexist by design, not by transition.

What narrows the claim

Three objections deserve to be taken seriously, and the first two do real work.

Observability isn't unlimited curiosity. A trace without a tenant identifier can never be sliced by tenant, however much of it you keep. You can only ask what the schema supports.

Correct, and it matters. Observability moves the binding decision from query time to instrumentation time — it does not abolish binding altogether. But the thesis concerns the class of intake, not completeness within it. "Everything, continuously" names a ceiling set by the sensor set in use; improving that set is exactly the scale axis already conceded. No further class of intake exists beyond continuous, provenanced observation — that is the claim. That some instance of it saturates its own ceiling is not.

Total retention is economically impossible. Storage grows superlinearly with traffic, and every mature system eventually reintroduces sampling and expiry. Observability degrades back into monitoring under budget pressure.

The economics are real, and the drift back toward aggregation is a genuine, common failure. But the category is defined by revisability and provenance, not by literal universal retention. A system that samples one request in a hundred while recording that it sampled, and can revise a belief when a new stream contradicts an old one, is still doing something a frozen corpus structurally cannot do. Cost decides where a system sits on the axis. It does not create a further position past the end of it.

Observability is correlational. A live system, queried indefinitely, still may not reveal why it failed — causal knowledge requires intervening, not just watching. So continuous intake is not the end of epistemics; deliberate experiment is a separate capability entirely.

This is the strongest objection, and it concedes something structural: passive observation underdetermines causal structure, a point Pearl's causal hierarchy makes precise. But intervention is a new source of evidence, not a new class of it. A chaos experiment, a canary release, a randomised rollout — each produces a stream, and continuous intake absorbs it, tagged as manipulated rather than merely observed. Provenance carries that tag. Intervention sits alongside continuous intake; its results still arrive through it.

What this establishes is narrow and should stay narrow. It says that continuous, provenanced, revisable observation is where the intake axis ends, because its complement is empty. It does not say any real system achieves this fully, that instrumentation choices stop mattering, or that observing more replaces the need to intervene. Those are separate claims, and weaker ones than this page is willing to make.