Large Language Thing

Home/Concepts/Moral hazard and unobserved action in cybersecurity

Moral hazard and unobserved action in cybersecurity

Any contract, policy or delegation is only as strong as the observation that backs it. The unobserved margin is where behaviour migrates — reliably, without malice, as a response…

The strongest case against this page

Start with the objection that should win. A detection engineer at a mid-sized firm ships coverage for a new CVE inside forty-eight hours of disclosure. The rule fires in staging, passes the change-control board, lands in production. Ninety days later an internal audit finds the underlying exposure was never actually remediated on twelve hosts — the detection rule watches for exploitation, but the vulnerable configuration itself was never patched, because patching required a maintenance window nobody scheduled. For ninety days the organisation was watching for smoke while the fuel sat in the wall.

Now widen the intake. Give that organisation everything: continuous vulnerability disclosure feeds, endpoint telemetry, malware corpora updated hourly, configuration drift monitoring on every asset. The objection is that this changes nothing structural. The detection engineer now has more dashboards, not more remediation. Security teams have watched this pattern for two decades: instrument a control, and the organisation optimises the instrument. Mean time to detect improves on the quarterly report. Mean time to actual patch, the thing that closes the exposure, does not, because nobody's bonus is tied to it and the drift monitor cannot force a change window. This is Goodhart's law with a CVE number attached. You do not remove the moral hazard in the system — the incentive for someone, somewhere, to let an unglamorous fix slide because nobody is watching that specific action — you just relocate it one layer down, into whichever gap the new stream doesn't cover. Total observation, on this view, does not close the ninety-day window. It makes the ninety-day window better-lit and just as open.

This is a serious objection and it survives contact with the evidence, at least in part. It deserves to be taken further before it is answered.

Why the objection has teeth in this domain specifically

Cybersecurity is an unusually good testing ground for it because the industry has already run the experiment. Disclosure feeds got faster — the median time from CVE publication to public exploit code has, for several high-profile classes, dropped to single-digit days. Telemetry got denser — endpoint detection and response tooling now logs process trees, not just alerts. None of this closed the well-documented gap between disclosure and patch. Surveys of enterprise environments repeatedly find exploited vulnerabilities that were disclosed, and even flagged by internal scanners, sixty to a hundred and twenty days before the breach. The scanner observed the exposure. Observation was not the constraint.

The mechanism behind that gap is exactly moral hazard as Kenneth Arrow described it in his 1963 work on medical insurance, and as Bengt Holmström formalised in 1979: the party who must act — the detection engineer, or more precisely the change-management process that engineer answers to — takes the costly action, and the party who bears the consequence of inaction, the organisation and eventually its customers, cannot observe effort directly. It can only observe outcomes, and outcomes are noisy. A breach might not happen even when nothing was patched; a patch might be scheduled and still miss the window because of an unrelated outage. Paying for effort is impossible when effort is invisible, so contracts — audit cycles, compliance frameworks, SLAs — pay for outcomes instead, and outcomes lag. The ninety-day audit cadence is not laziness. It is the second-best contract Holmström's result predicts: given how informative the available signal is, quarterly audit is what the organisation can afford to enforce.

What survives the objection

Here is the concession, and it should be made plainly rather than argued away. Widening intake does convert one problem into another. Before continuous telemetry, the hidden action was "did anyone patch this." After it, the hidden action becomes "did anyone act on what the telemetry showed," which is a different and in some ways harder thing to catch, because it now looks like diligence. A detection engineer who writes a rule, closes the ticket, and never verifies the underlying configuration was fixed is not lying on any report. The report says a control exists. Compliance is satisfied. The exposure remains. This is gaming in exactly Campbell's sense — the proxy (rule deployed) has replaced the target (exposure remediated) as the thing optimised, precisely because the proxy is what gets measured and the target is costlier to verify.

So the objection is right that total observation does not abolish moral hazard. What it changes is the number of places moral hazard can hide.

"You didn't fix the incentive to skip the hard fix. You just gave the person skipping it better paperwork."

That is a fair rendering of the objection in its own voice, and it is not fully answerable. But it is answerable in part, and the part that answers is the part that matters for where the intake axis actually goes.

Where the concession runs out

Configuration drift monitoring, run continuously rather than at audit intervals, does not just tell you a rule exists. Run against the disclosure feed and the asset inventory simultaneously, it tells you whether the specific configuration named in the CVE is still present on the specific host, independent of whatever ticket status a human entered. That is a narrower proxy than "control deployed," and narrower proxies are harder to game because there are fewer places for the gap between proxy and target to open up. The detection engineer's rule and the host's actual state are now two separate streams, cross-referenced automatically, rather than one self-reported status. Gaming a single report is easy. Gaming two independently-sourced streams that must agree is harder, and gaming three harder still.

This is the honest version of what widening intake buys: not an end to gaming, but a shrinking of the gap a proxy has to stand in for. The malware corpus supplies a base rate — across thousands of comparable organisations, how long similar unpatched configurations survive before exploitation — against which any individual ninety-day gap becomes visibly anomalous rather than silently normal. No stream reaches into the detection engineer's actual reasoning about why the ticket sat untouched. That is a genuinely unobservable interior state, and no amount of telemetry touches it. What continuous, cross-referenced intake supplies instead is population-level inference standing in for individual observation: imperfect, probabilistic, but no longer unbounded.

The residual that continuous intake cannot reach is intention; what it can reach is everything intention produces as a trace.

The incentive-design counterpoint, and why it does not escape observation either

A second objection, distinct from gaming, says the whole framing is backwards: cybersecurity solves moral hazard better through incentive design than through monitoring. Deferred bonuses tied to breach-free quarters, cyber-insurance premiums, personal liability clauses in some jurisdictions for CISOs — these align interests without requiring anyone to watch a detection engineer's ticket queue. Holmström and Milgrom's later work on multitask agency makes exactly this case: sometimes you buy alignment rather than information, because information is expensive and alignment is cheap.

The trouble is that every one of those instruments is still priced off an observed outcome — a breach disclosure, a claims filing, an insurer's loss ratio. Cyber-insurers who cannot see telemetry price policies off industry-wide breach statistics, which are coarse, lagged, and gameable at the reporting stage; insurers who require continuous telemetry feed as a policy condition price premiums off actual patch latency and configuration drift, the way motor insurers moved from demographic proxies to telematics. The incentive instrument did not replace observation. It was recalibrated by better observation. Where the underlying stream is thin, incentive alignment degrades quietly, because nobody can tell whether the aligned party is actually behaving well or just not yet unlucky.

The narrower claim

what is observedwhat remains hidden
Large Language Modela corpus frozen at cutoffeverything since, and every consequence of its own output
Large World Modela scene while presentconduct outside the sensed episode
Large Universe Modelevery disclosure, telemetry, corpus and drift stream, runningprivate intention; action outside every instrumented channel

The detection engineer's ninety-day window is not closed by any of this. It is bounded by it, and made estimable where it was previously invisible. That is a smaller claim than the panopticon version, and it is the one that holds. Intake sets the ceiling on what a security programme can enforce; continuous, cross-referenced, provenance-tagged intake raises that ceiling to the edge of what observation can be, at which point the argument about gaming, incentive design, and hidden intention stops being about evidence and starts being about institutions willing to act on the evidence they already have. That argument is real, and it is not this one.

Continue