Propositions

Three deployments in three unrelated sectors published the side of their error rate that made them look effective and omitted the side that carried the cost.

A children’s hospital reported that its vendor-built sepsis model caught a given share of cases and could not report the positive predictive value, because the vendor had never published one. A disk-drive factory reported catching fifteen leaks out of sixteen and never counted the false alarms. A pharmacy chain ran face matching across hundreds of stores for eight years and counted nothing at all — which is the same omission with the measurement apparatus removed.

The direction is consistent. Recall flatters a detector; precision is what its false positives cost somebody else. In all three the published number was the one that described the system’s successes, and the missing number was the one that would have described what it did to the people it was wrong about.

The counter-case shows the omission is a choice rather than a constraint. Amsterdam measured false positive shares for eight demographic groups across three datasets, before and after a debiasing intervention, and compared them against the human process it was replacing. That is both sides of the error rate, disaggregated, on a live system — and it is what let the city discover that its correction had moved the harm rather than removed it. Nothing about the domain made this easier than sepsis alerts or shoplifting detection. Somebody decided to count.

A national grid operator cut its solar forecasting error by two thirds and never measured the balancing cost the forecast existed to reduce, converting it instead through a planning rule of thumb. A government chatbot raised its self-assessed accuracy from 76% to 90% and moved user satisfaction not at all. A sepsis alert improved time to antibiotics while mortality, ICU admission and ICU-free days stayed flat and underpowered. A support copilot measured resolutions per hour; the randomized replication that followed it measured customer retrials too, and found those unmoved.

A seven-day forecast of urgent dialysis demand was run live twice at four Toronto hospitals and measured on one quantity only — mean absolute error in procedures per day. Nurse hours, backfill decisions, overtime and cost were the reason the forecast existed and none of them was measured; the paper closes by saying whether the model would save anything is an open question. The staffing estimate that did exist survives only in a trade-press report of a conference talk given eight months before publication.

In each, the measured quantity was the tractable one and the quantity that justified the spend went unmeasured. The pattern is not dishonesty. It is that proxies are cheap, immediate and attributable, while outcomes are slow, confounded and owned by somebody else in the organisation.

The counter-case is the one deployment here where somebody measured the outcome properly, and it shows why the proposition matters rather than refuting it. A randomized trial of AI chest X-ray prioritization measured both halves: the proxy the feature was built to move — time from X-ray to report — improved significantly, from 47 hours to 34.1. Every clinical outcome it was installed to improve stayed flat, in an adequately powered trial with tight intervals. The proposition’s falsifier asks for a case that measured both and found them to move together. This one measured both and found them to move apart.

The failure mode it produces is specific: a deployment can report a real, large, correctly measured improvement and still have no evidence that it did the thing it was bought to do.

The closest any case comes is a national grid operator that disclosed £1,038,500 of development cost and roughly £45,000 a year to run the service — and then derived its benefit by multiplying a planning rule of thumb.

Elsewhere the halves separate cleanly. A peer-reviewed field study establishes a 15% productivity effect and discloses no cost anywhere in the record, because the firm is anonymous. A rideshare company reports per-engineer AI spend to the dollar on an internal dashboard, against an executive who said publicly that he could not draw a line from the tools to a shipped feature. A factory claims millions of dollars a year of avoided scrap with no derivation. A government team publishes two and a half years of methods and not one cost figure.

The sharpest instance is a city that measured the benefit and found it negative. Amsterdam’s welfare fraud model was projected in internal documentation to keep up to 125 people out of debt collection and save €2.4 million a year. Its councillors were never told what it cost; the figure exists only because three newsrooms asked, and the city’s answer — about €500,000 plus €35,000 for a contractor — came with a caution that it was an estimate. The measured result was more investigations and no improvement in the hit rate.

Cost telemetry and value telemetry arrive years apart, and cost arrives first. That ordering is itself the finding: metered tools make spending legible immediately, while the benefit needs an experiment almost nobody runs.

One case now breaks that ordering without breaking the proposition. In AAI-2026-021 somebody did run the experiment — a randomized trial across five NHS trusts, powered, pre-registered, independently analysed — and measured the benefit to a confidence interval. There was none. The cost half is registered as that trial’s seventh secondary outcome and has not been published; the paper calls the costs of the function “considerable and avoidable” and gives no figure. So the halves still arrive separately, and for once it is the benefit that arrived first. The proposition survives its most rigorous test to date, which is not the same as being vindicated by it.

Dutch officials reviewing students the algorithm had scored identically selected those with a non-European migration background 5.5 times as often. The reviewer amplified the model rather than checking it, and nobody measured that for eleven years.

Rite Aid’s staff were handed a match alert with no confidence estimate, no training in how the system worked, and an instruction to act on it. In the sepsis deployment the human never entered the loop at all: workflow exclusions suppressed the alert in roughly four cases out of five before anyone saw it, which is why the delivered sensitivity was 18% against a vendor-reported 91%.

The counter-case matters as much as the three. A factory leak detector emails a probability and its supporting evidence to an engineer, who walks to the machine and physically checks; nothing on the line is actuated automatically. That arrangement holds — and what makes it hold is not that a human is present but that the human’s job is defined, bounded, and verifiable against the physical world.

Cruise briefed three regulators in person on the morning after its vehicle dragged a pedestrian, and was charged over neither meeting. The criminal count came from two written incident reports filed under a standing order — the 1-day report and the 10-day report, the documents nobody staffed.

The Dutch regulator found that failing to justify the selection criteria was itself the violation, independent of the disparity those criteria produced. In the Workday litigation, discovery rules and attorney–client privilege — not model documentation — decide what anyone will ever learn about the system’s disparate impact. Korea’s regulator reached a privacy policy that named some overseas processors and omitted one.

The clearest instance is the one where no model was ever identified. Across more than two thousand court decisions about fabricated legal authority, a specific AI product is named in about one row in nine — and in the Nebraska Supreme Court decision read for that case, counsel denied using AI at all. The court struck the brief, dismissed the appeal and referred counsel for discipline anyway, on the duty of candour and the fictitious citations. The sanction did not depend on establishing a cause.

Italy’s Garante reached DeepSeek in three days without looking at the system at all. Its findings were that the privacy policy was English-only and inadequate on its face, that no lawful basis was stated for each processing activity, that the policy itself disclosed storage in China without the Regulation’s safeguards, that no EU representative had been appointed, and that the reply to the Authority’s questionnaire failed to clarify the processing — a separate breach, of the duty to cooperate. Every one of those is readable from published documents and correspondence by someone with no access to the service.

In none of these did an authority examine the model and find it wanting. The model was the occasion; the document was the offence.

In the only randomized trial in this library, experienced maintainers working on their own repositories were 19% slower with AI assistance and believed they had been 20% faster. That is an error of direction, not magnitude, held by people with direct hands-on experience of the tool during the task being measured.

Expert forecasting did worse. Economists and machine-learning researchers predicted speedups of roughly 38 to 39% for the same trial.

The government chatbot shows the same gap from the user’s side: people discounted explicit inaccuracy warnings because the answers carried the GOV.UK brand, and the team recorded the finding and published it. Fourteen points of measured accuracy later, satisfaction had not moved.

The pattern recurs where the measurement is most careful. Twenty-two public service broadcasters had 271 journalists rate more than 2,700 AI assistant answers about the news and found a significant issue in 45% of them; the same report cites separate research finding that just over a third of UK adults completely trust AI to produce accurate summaries, rising to almost half of under-35s. Measured error and reported confidence point in opposite directions in the same document.

What belief tracks, in all of them, is fluency — the system’s outputs are well-formed, arrive quickly, and look like the thing that was asked for. None of those properties is the effect.