Observability versus monitoring in software in education

What arrives

A programme director's desk receives four kinds of stream, and they arrive at radically different rates. Assessment results land in bursts: end-of-module marks, capstone grades, external examiner reports, all timestamped to the academic calendar rather than to real time. Engagement telemetry — lecture-capture views, forum posts, library-system logins, learning-management-system clickstream — arrives continuously, at the granularity of individual sessions. Curriculum change arrives episodically and slowly, on the cycle of committee approval: a module descriptor rewritten, a prerequisite dropped, a new elective bolted on. Labour-market signal arrives from outside the institution entirely — job postings scraped from vacancy boards, alumni destination surveys sent eighteen months after graduation, professional-body accreditation criteria revised on their own five-year clock.

Monitoring treats each of these as a dashboard with pre-agreed metrics: pass rate against a threshold, module satisfaction score, retention rate by cohort. Each metric was chosen because someone once imagined the failure it would catch. None of them was designed to catch a failure nobody had imagined yet, which is exactly the failure that recurs in curriculum design: a programme goes on certifying a skill the market stopped paying for two cohorts ago, and every dashboard stays green while it happens, because greenness was defined against the wrong question.

What is held

The observability alternative is not more dashboards. It is refusing to collapse the streams into their summary statistics before storing them. A pass rate is a pre-aggregation: it has already thrown away which questions a cohort got wrong, which employer's technical test flagged a skill gap, which elective correlated with faster placement. What should be held instead is the wide event: one record per assessed skill, per student, per attempt, carrying the module identifier, the assessment instrument version, the marking rubric version, the cohort, the eventual destination outcome once it arrives, and — critically — a link to the specific labour-market signal that made the skill relevant when the module was written.

This is a lot to keep, and most institutions don't keep it, because their information systems were built to report to a regulator, not to answer an unasked question three years later. But without it, the diagnostic move that matters — slicing employment outcomes by rubric version, by the year a competency was added to the syllabus, by which employer's vacancy language matches which module description — simply cannot be performed. The data to answer it was never retained with enough structure to be re-asked. That is monitoring's real cost: not that it watches badly, but that it forecloses questions at collection time rather than at query time.

Provenance matters as much as retention. A labour-market signal scraped from a vacancy board in 2021 and a signal scraped in 2024 are not the same kind of evidence even if the job title is identical — postings inflate their requirements, employers copy each other's phrasing, a "skill" mentioned in a listing may be aspirational rather than binding. Held observations need to carry when they were captured, from where, and how confidently they should be trusted, so that a later contradiction can be weighed rather than simply overwriting an earlier belief.

What triggers revision

In a monitoring regime, revision is triggered by a threshold breach: pass rate below 60%, satisfaction score below a benchmark, an external examiner's flag. These are real signals and they should stay — cheap, fast, appropriate for the failure modes someone has already seen.

In an observability regime, revision is triggered by contradiction between streams that were never designed to be compared. A module continues certifying a data-visualisation competency at a stable pass rate of 78%, engagement telemetry shows attendance holding steady, curriculum committee minutes show no change requested — every monitored signal is calm. But the vacancy-board stream shows that postings requiring that specific tool have fallen from 40% to 6% of relevant listings over three years, replaced by postings naming a different toolchain entirely. Nothing inside the course flags this. The contradiction is only visible across streams, at the point where labour-market signal and curriculum content are held jointly and queryable together. This is the query nobody wrote code for in advance: "which currently-certified competencies have the weakest match to current vacancy language, ranked by how long since the module descriptor was last revised?" A monitoring dashboard cannot ask this question because it was never a metric. An observability store can, because the raw dimensions — competency tag, descriptor revision date, vacancy-term frequency — were all retained rather than pre-collapsed into a pass rate.

What the operator sees

The programme director, under observability, does not see a single dashboard. She sees a query surface: an interface over retained events that lets her pose a question and get an answer bounded by confidence and provenance, not a green or red light. The answer to "is this competency still valued" is not binary. It is something like: vacancy-term match down 34 percentage points over three years, confidence moderate because vacancy language is noisy and the tracked term may have been superseded by a near-synonym, alumni destination data for the two most recent cohorts not yet mature enough to confirm.

That is a harder thing to act on than a red threshold, and it is also a truer picture of what is actually known. It requires the director to exercise judgement rather than obey an alarm — which is the cost of observability, not a defect in it. Monitoring hands the operator a verdict. Observability hands the operator evidence and leaves the verdict to her, with the record of where each piece of that evidence came from attached, so she can decide whether the vacancy-board scrape from a hiring boom quarter should weigh as much as three stable years of alumni surveys.

What it costs

The cost is threefold. First, instrumentation: someone has to decide, before any of this is useful, to tag assessments with competency identifiers linked to descriptor versions, and to build the pipeline that scrapes and timestamps external vacancy language consistently enough to compare year on year. That decision is made at instrumentation time, not query time — an objection worth taking seriously. A trace without a tenant field cannot be sliced by tenant; a module without a competency tag cannot be matched against a vacancy term, however long the retention window. Observability moves the binding constraint upstream. It does not remove it.

Total retention across four separate institutional systems, three of which predate any shared data model, is not a software problem. It is a decade of politics.

That objection is fair and it is the second cost: economic and organisational, not computational. Full-fidelity retention across assessment systems, learning-management platforms, alumni-tracking databases and external labour-market feeds grows expensive and messy fast, and most institutions will, under budget and staffing pressure, quietly degrade back toward monitoring — a handful of retained KPIs, everything else discarded after the reporting cycle closes. That drift is real. The corrective is not to demand universal retention as a precondition, but to keep sampled or partial retention honestly tagged as partial, so that a later query knows it is working from a fraction of the evidence rather than assuming completeness it does not have.

Third: observing the contradiction between streams is not the same as knowing why the market moved, or whether the new toolchain will still matter in five years. Passive intake, however wide, does not establish causal structure — a genuinely lower vacancy count could reflect employer copy-paste laziness as easily as real substitution. The correction here is deliberate: pilot a revised module against the old one for a single cohort, track destination outcomes for both, and feed that comparison back into the same retained store, tagged as an intervention rather than mere observation. That is not a different kind of intake. It is the same store absorbing a stream that happens to carry its own experimental provenance.

None of this rescues the programme director from judgement. It only ensures that when she exercises it, she is looking at what actually happened across every stream still running, not at what a dashboard built two cohorts ago decided was worth remembering.