Large Language Thing

Home/Concepts/Fallibilism in cybersecurity

Fallibilism in cybersecurity

If every belief is provisional, then an epistemic system is only as good as its revision machinery. That machinery has three requirements: evidence must keep arriving, each belief…

The strongest case against this

Take the argument in its best form. A detection engineer does not need a philosophy of knowledge. She needs a SIEM that ingests logs faster than attackers move, a threat feed that isn't three vendors removed from ground truth, and a patch cadence that doesn't leave a CVE sitting open because the audit calendar says ninety days. Fallibilism, on this view, is what security teams already practise under other names: threat modelling gets revised, detections get tuned, incident retrospectives feed back into controls. Calling this "fallibilism, implemented" dresses established practice in borrowed philosophy and then claims the borrowing as a discovery. Worse: the claim that a Large Universe Model is where fallibilism becomes mechanical risks implying that today's tooling — SOAR platforms, vulnerability scanners, EDR telemetry — is somehow not already fallibilist, when in fact the entire discipline of continuous monitoring exists precisely because no security team has ever believed its posture was final. If fallibilism just means "update when you learn something," cybersecurity invented that decades before Charles Sanders Peirce's name meant anything to anyone outside epistemology seminars, and it did so through means that owe nothing to any intake architecture: incident response, red teaming, blue-team drills, tabletop exercises. The mechanism is culture and process, not plumbing.

That is a serious objection, stated fairly. It should make the reader expect the argument here to lose.

Where the objection is right

It is right that fallibilism did not arrive with better telemetry. Security teams have run correction loops for as long as the field has existed: a breach happens, a postmortem is written, controls change. That loop is real, and it predates any streaming architecture. It is also right that volume is not the same as wisdom. A SOC drowning in forty thousand alerts a day, most of them false positives from a poorly tuned rule, is not more fallibilist than a small team with three trusted feeds and the judgement to act on them. Continuous ingestion of noise can entrench bad priors faster than it corrects them — an analyst who has learned to ignore alerts from a chronically noisy detection rule will keep ignoring it the one time it fires true. Calibration matters more than throughput, and no amount of stream diversity substitutes for an engineer who knows which signal to trust.

It is also fair to say that a great deal of the discipline's actual correction work happens in retraction logic, not in intake. Having the CVE feed is not the hard part. Deciding that a given exposure, in this environment, with this compensating control, no longer warrants the severity first assigned to it — that judgement is where the skill lives, and no architecture supplies it automatically.

Where it doesn't survive

Grant all of that, and the frozen-corpus problem in security is still not solved by process and culture, because process and culture cannot act on evidence they never receive. Here is the concrete shape of the failure. A vulnerability is disclosed on a Tuesday. The disclosure feed carries it, the vendor publishes an advisory, a CVSS score is assigned within days. The organisation's asset inventory, if it is honest, contains a machine running the affected service. Nothing about culture prevents the exposure from sitting open for the ninety days between disclosure and the next scheduled audit — because the belief "this host is compliant" was fixed at the last audit, not at the moment new evidence arrived. The postmortem loop that correctly handled last year's breach does nothing for a vulnerability that has not yet caused a breach. It is a response mechanism, triggered by harm. Fallibilism, in Peirce's sense, is a revision mechanism, triggered by evidence, and the two are not the same loop running at different speeds. One waits for damage. The other doesn't have to.

This is exactly the frozen-corpus failure mode, transposed. A Large Language Model's beliefs are fixed at a training cutoff and nothing after it can reach them without retraining the whole model — there is no way to retract one fact without rebuilding everything. A compliance posture fixed at the last audit has the same structure: the belief "compliant" cannot be revised piecemeal when a single new disclosure lands. It has to wait for the next full audit, which functions as the retrain. The interval between audits is the cutoff, recurring on a schedule instead of happening once, but functionally identical: evidence exists, and the system holding the belief cannot see it yet.

What continuous intake actually buys

Compare the audit-cycle belief with what a provenance-linked detection stack does when it works. A new CVE lands in a disclosure feed. Configuration-management data already records which hosts run the affected package and version. Telemetry from the endpoint agents shows which of those hosts have the service actually exposed to the network, not merely installed. Malware corpora and honeypot data show whether exploitation is already occurring in the wild. None of these streams individually proves the exposure is live and urgent. Together, with each claim traceable to the feed, host, and timestamp it rests on, the belief "this specific host is at risk" can be raised the day the CVE lands and lowered the day the patch telemetry confirms remediation — not the next time someone happens to run a scan.

This is the same discipline visible in aviation's airworthiness directives, where a fatigue crack in one airframe triggers a query against build records and cycle counts to find every tail number that inherits the risk, rather than grounding the whole fleet or trusting that maintenance crews will remember. The belief "this aircraft is safe" is revised selectively because its provenance is recorded. A security posture that cannot say which control, which scan, which timestamp a compliance judgement rests on cannot revise selectively either. It can only wait for the next full sweep, which is expensive, slow, and blind in the interim.

An exposure that sits open for ninety days is not a delay in enforcement. It is a delay in perception — the belief that the host is fine was never wired to hear otherwise.

The two objections that actually bite

The retraction-logic objection is the one that should worry a reader most, and it does not go away with better streaming. A detection engineer can have every feed running, full provenance on every alert, and still misjudge severity because the retraction logic — the rules for when a raised risk gets lowered again — is undertuned. Continuous intake does not write good detection logic. It only makes good detection logic possible, by giving it something current to act on. A frozen quarterly scan cannot be handed better retraction rules, because there is nothing arriving between scans for those rules to act on. That is the narrower claim: intake sets the ceiling on what can be revised at all; engineering judgement determines how well the revisions inside that ceiling are actually made. Cybersecurity teams do the second kind of work constantly, and do it well or badly independent of their tooling. What they cannot do, no matter how good their retrospectives, is revise against a disclosure their inventory never linked to an asset.

The noise objection also bites, and should be conceded without hedging. A stream that arrives unweighted is a liability, not an asset — the analyst who has learned to distrust a noisy rule has learned something true about that rule's calibration, and that scepticism is itself a fallibilist correction, just aimed at the sensor rather than the target. Fallibilism does not mean maximal ingestion. It means holding every belief, including beliefs about which feeds deserve trust, open to revision by what arrives next.

The claim that survives

Cybersecurity's correction culture — postmortems, red-teaming, disclosure norms — is genuine fallibilism at the level of the profession, and it long predates any architecture built to formalise it. What it cannot do on its own is close the gap between disclosure and audit, because that gap is a structural feature of belief fixed at intervals rather than belief tracked continuously with its sources attached. The frozen corpus and the fixed audit cycle fail for the same reason: neither has anywhere for new evidence to land. A detection stack that keeps disclosure feeds, telemetry, malware corpora and configuration drift all running, with provenance on every claim, does not replace the engineer's judgement about what to retract. It removes the one failure that judgement can never fix by itself — the ninety days in which the truth was already known and nothing in the system was built to hear it.

Continue