The developers who were sure AI made them faster

Case in one sentence

Sixteen experienced open-source maintainers were randomly assigned to complete some of their own repository issues with AI tools and some without, took 19% longer on the ones where AI was allowed, and still believed afterwards that AI had sped them up by 20%.

Executive summary

Between February and June 2025 the nonprofit METR ran a randomized controlled trial on real software work. Sixteen developers — typically over a decade of experience, an average of five years and 1,500 commits on the specific repository they worked in — supplied lists of genuine issues from their own projects. Each issue was randomized to AI-allowed or AI-disallowed after it had been defined and after the developer had forecast how long it would take under each condition.¹

Allowing AI increased completion time by 19%.¹ The interval around that estimate runs from +2% to +39%, a number METR published seven months later rather than in the paper.²

The more durable finding is the gap between that measurement and everyone’s belief about it. Before the trial the developers forecast a 24% speedup. After completing it — after months of using the tools on their own code and being slowed down — they estimated a 20% speedup. Economists forecast 39%; machine-learning experts forecast 38%.¹ Every group, including the people who lived through the experiment, was wrong about the sign.

The case does not end there, and the ending is the part that generalises. METR started a second, larger experiment in August 2025: 57 developers, 143 repositories, more than 800 tasks. In February 2026 it published the result as a methods failure. Developers had begun refusing to work without AI, and 30% to 50% of participants said they were withholding tasks they did not want to attempt unassisted. METR judged its own central estimate “a bad proxy” and announced it was redesigning the study.² The measurement instrument had been broken by the thing it was measuring.

Research question

When experienced engineers use AI tools on code they know deeply, does the time they spend actually fall — and if it does not, do they notice?

Organization and operating context

METR is a research nonprofit that measures frontier AI systems’ capacity to perform complex tasks autonomously.¹ It is not the deploying organization in this case; the deployments are sixteen individual maintainers’ own working practices, which is why this case sits in a library otherwise made of corporate deployments.

The subjects are the operating context. The repositories average 23,000 stars, 1,100,000 lines of code, 20,000 commits, 710 committers, and roughly ten years of age, and they hold contributions to high review standards.¹ The developers were regular contributors to them, averaging five years on the repository — 59% of that repository’s lifetime.¹ They were not novices being handed a tool; they were the people who wrote the code the tool was being asked to change.

Their AI experience was real but uneven: 93% had used LLM tools such as ChatGPT, and 44% had used the Cursor IDE the study provided.¹

The situation before AI

The pre-AI baseline is simply how these maintainers worked: reading their own codebase, writing code, testing and debugging, and submitting pull requests through their project’s normal review process. The screen-recording labels make the composition visible — in the AI-disallowed condition, a larger share of time goes to actively writing code and to reading or searching for information.¹

The relevant prior belief was industry-wide and confident. Earlier controlled experiments had found large speedups: Peng and colleagues measured a 55.8% faster completion with GitHub Copilot, on a task specified as implementing an HTTP server in JavaScript.³ METR’s contribution is not to dispute that number but to change the task: real issues, in a repository the developer maintains, judged by that repository’s own reviewers.

The AI intervention or event

Developers listed real issues they wanted to fix, capped at roughly two hours each, and forecast each one’s duration both with and without AI. Only then was each issue randomized by simulated coin flip.¹ That order matters: because the work was fixed before the treatment was known, the trial measures time to do a defined piece of work rather than volume of output, which is what makes it immune to the criticism that AI simply produces more verbose code.¹

In the AI-allowed condition developers could use anything, including nothing; most used Cursor Pro with Claude 3.5 or 3.7 Sonnet, and METR provided subscriptions and basic training. In the AI-disallowed condition no generative tooling was permitted, though ordinary search engines remained available.¹ Developers recorded their screens, and self-reported time before and after pull-request review.

Then, in August 2025, METR ran it again with 57 developers and over 800 tasks, at $50/hour instead of $150.²

Outcomes and economics

The 2025 trial. Developers completed 136 AI-allowed and 110 AI-disallowed issues.¹

All rows are from the paper except the interval.¹

Where the time went, on the labelled subset: with AI allowed, developers spent proportionally less time actively coding and less time reading or searching, and more time prompting, waiting on generations, reviewing AI output, and idle.¹ Roughly 9% of time went to reviewing and cleaning AI-generated code and about 4% to waiting for generations.¹ Developers accepted fewer than 44% of Cursor’s generations; 75% said they read every line of AI output, 56% said they often had to make major changes to clean it up, and 100% said they had to modify it at all.¹

The 2026 follow-up. For the ten developers carried over from the original study, the estimate flipped to an 18% speedup, interval −38% to +9%. For newly recruited developers it was a 4% speedup, interval −15% to +9%.² Both intervals cross zero. METR does not present these as findings.

The 2026 survey. Across 349 technical workers surveyed February to April 2026, the median self-reported change in the value of work due to AI was 1.4x to 2x; the median self-reported change in speed was 3x. Respondents retrospectively put themselves at 1.3x in March 2025 and forecast 2.5x for March 2027. METR staff reported the lowest gains of any subgroup studied.⁴

No cost figures are reported for any organization. What the trial prices is participant time at $150/hour, which is an experimental cost, not a deployment cost.

Causal assessment and competing explanations

This case is labelled causal, and it is the library’s first. The justification is narrow and worth stating precisely: treatment was randomly assigned to units of work that were fully specified before assignment, so within this sample, the 19% figure is an effect of allowing AI and not a selection artifact. That is a stronger design than anything else in this library.

The label does not travel. It licenses “AI allowed slowed these developers on these issues”, not “AI slows developers”.

Four competing explanations deserve to be named.

Small sample, unusual people. Sixteen developers is few, and they are not a random draw from anywhere. METR’s answer is that treatment was randomized at the issue level, giving 246 units, and that clustered and bias-corrected standard errors give similar results while a hierarchical bootstrap gives wider ones.¹ The blog post concedes those clustered errors were “not reported in the released paper, but forthcoming”.⁵

The experiment made people use AI more than they would. METR finds developers who reported experimenting or maximising AI use were slowed more than those who used it normally — so this inflates the estimate but does not create it.¹

Self-reported time is unreliable. Recomputing from screen-recording duration on the same subset gives 25% against 24%.¹ This one is answered.

It is about these repositories and these people. METR agrees, and this is the explanation with real force. Developers were slowed more on issues where they had high prior exposure, and developers who forecast a large speedup for a particular issue were not slowed on it.¹ One developer with more than 50 hours of Cursor experience showed a speedup. The authors state their results are consistent with greenfield projects or unfamiliar codebases seeing substantial speedup.¹

Other trials find a speedup, so one of them must be wrong. They need not be. The most directly comparable one is a randomized trial at Google: 96 full-time engineers, randomly assigned, instrumented time on a single enterprise-grade task inside Google’s own infrastructure in summer 2024. It found the AI arm finished in 96 minutes against 114, a difference significant on an unadjusted test of log time (p = .038), and estimated the adjusted effect at about 21% faster.⁶ Set beside METR, the two are a near-perfect illustration of the moderator METR names rather than a contradiction of it: Google’s participants worked one designed task — a pre-existing change list of ten files and 474 lines — with a mean completion time of 104 minutes, while METR’s worked real issues they had chosen in repositories they maintain. Unfamiliar, bounded, specified work against familiar, open-ended, self-owned work. METR’s own statement that its results are consistent with substantial speedup on greenfield or unfamiliar code predicts the Google finding.

Two further things about that paper are worth recording rather than passing over, because this case is partly about how productivity numbers are built. The 21% comes from the adjusted model, whose estimate is not significant at the conventional threshold — β₁ = −0.24, 95% CI [−0.51, 0.03], p = 0.086 — while the significant result is the unadjusted t-test, which the paper reports without a percentage attached. The abstract says AI “significantly shortened” time on task and gives “about 21%” in the next sentence. Each statement is defensible alone; together they read as a significant 21%, which is not what the regression table shows.⁶ And the paper describes itself as a “lab study” in the same abstract whose title calls it “enterprise-based” — a candour that most readers of the headline figure will never reach.

The perception gap is a separate finding with a separate basis: it is a comparison of elicited forecasts against measured times, and it does not depend on the causal design at all.

Failures, limitations, and governance

• Experience did not protect against the misjudgment; it may have caused the slowdown. The developers most likely to be wrong about the tool’s value were the ones who knew the code best.
• The tools were used as intended and still cost time. The mechanism is not misuse. It is review burden: under 44% acceptance, 100% of accepted output requiring modification, and 9% of working time spent cleaning up.¹
• Post-hoc estimates did not converge on the truth. Living through the experiment moved the developers’ estimate from 24% faster to 20% faster, in the wrong direction from a measured 19% slower.¹
• The measurement broke. By late 2025, 30% to 50% of participants were withholding tasks from randomization because they did not want to do them without AI, and recruitment suffered for the same reason. METR concluded its estimate was likely a lower bound on true speedup and abandoned the design.²
• A pay cut confounds the follow-up. METR reduced participation pay from $150 to $50 an hour and says this likely contributed to the selection problem, so the two datasets differ in more than their dates.²
• The headline uncertainty was published late and elsewhere. A reader of the paper alone cannot state the interval on its central claim.
• Everything here comes from one organization. Two experiments and a survey, three data collections, one team.

What this case demonstrates

1. Under randomization, allowing early-2025 AI tools made experienced maintainers of large mature repositories slower at defined issues in their own code, by 19% with an interval of +2% to +39%.
2. Self-assessment of AI’s effect on one’s own work can be wrong in sign, not merely in magnitude, and direct experience of the tool does not correct it.
3. Expert forecasting was wrong in the same direction and by more than the practitioners’ — economists and ML researchers predicted roughly 38–39% speedups.
4. Task realism changes the answer: the contrast between a 55.8% speedup on a specified synthetic task³ and a 19% slowdown on real issues in ten-year-old codebases is a fact about what is being measured.
5. The cost of AI assistance is concentrated in review and rework, and it is visible: acceptance under 44%, 9% of time cleaning output, 100% of developers modifying what they accept.
6. Familiarity is a moderator. Slowdown was larger where developers had more prior exposure to the task, and the tools help most where the developer knows least.
7. Rising adoption can destroy the ability to measure adoption’s effects. A randomized design that requires a no-AI arm becomes unrunnable once participants decline the arm, and the resulting bias runs in a known direction.
8. Self-report and measurement diverge systematically, which METR’s own 2026 survey concedes while nonetheless arguing surveys remain useful.⁴

What this case does not demonstrate

• It does not show that AI tools slow down software developers generally. METR says so explicitly, the moderators it identifies are absent from most software work, and a randomized trial at Google on a bounded enterprise task found the opposite sign.⁶
• It does not resolve the disagreement in the literature, and neither does the Google trial. The two differ in setting, task design, tools, year, and unit of randomization all at once, so nothing can be attributed to any one of those differences.
• It does not measure AI’s effect on work quality, only on time. The follow-up notes participants reporting quality differences between conditions that the design never captured.²
• It does not describe current tools. The trial ran on Cursor Pro with Claude 3.5 and 3.7 Sonnet, before the agentic tools that reshaped the follow-up.
• It does not show that developers are still slowed down. METR’s own reading in February 2026 is that developers are probably faster now, on evidence it calls very weak.²
• It does not establish the size of any 2026 effect in either direction. Both follow-up intervals cross zero, and METR disowns the estimates.²
• It does not tell an organization whether its AI spend pays back. There are no organizational costs or revenues anywhere in this record.
• It does not settle whether the perception gap persists. The 2026 survey measures perceptions with no counterfactual to check them against.⁴

Evidence assessment

Grade B — an unusually well-instrumented experiment with no chain outside the organization that ran it.

The 2025 trial is the best-designed piece of evidence in this library. Randomization after task definition removes the output-inflation problem that undermines most field studies of coding tools; the primary outcome was validated against an independent measurement of the same quantity; four alternative estimators and four uncertainty procedures are reported; twenty-one candidate explanations were assessed and the authors state which five they think contributed, which ten are unresolved, and which six they find evidence against.¹ The dataset is public. The authors’ conclusions are hedged more carefully than most reporting of them.

Three things hold it at B rather than A.

One institution. This case carries three source families — the 2025 trial, the 2025–26 follow-up, and the 2026 survey — and by the letter of the triangulation rule that is three chains. It is not three independent chains. All three were designed, run, analysed, and published by METR, two of them by overlapping authors, and ten developers appear in both experiments. Counting them as independent corroboration would be a category error, and this case does not. The corroboration that exists is internal: screen recordings against self-report, and estimator against estimator.

No peer review, and an interval published elsewhere. The paper was on arXiv and unreviewed when read. Its central estimate’s confidence interval is drawn in Figure 1 and stated nowhere in the text; the numeric bounds used here come from a blog post seven months later.² The authors also conceded in the accompanying blog that clustered standard errors were forthcoming rather than published.⁵ For a result whose whole force is quantitative, that is a real hole.

Recorded discrepancies. METR’s February 2026 post describes its earlier finding as a “20% slowdown” where the paper says 19%.² The v2 abstract counts 21 candidate factors where METR’s own blog summary says 20. Neither changes any conclusion; both are recorded rather than smoothed.

Conflicts run in both directions and are worth stating plainly. METR’s remit gives it an interest in evidence that in-the-wild capability lags benchmarks, and this result serves that interest. Against that: it published a null-signal follow-up that wrote off an expensive experiment, it reports that its own staff give the lowest AI-uplift estimates of any group it surveyed and offers “METR staff overindex on METR’s previous findings” among the explanations,⁴ and it released the data. That is not the behaviour of a team protecting a result.

Two sources here are contrasts, not corroborations, and neither moves the grade. The GitHub Copilot trial was read in abstract only, and three of its four authors worked for the vendor of the tool it evaluated.³ The Google enterprise trial was read in full and is the more useful of the two, because its design sits much closer to METR’s: a randomized trial, instrumented time, professional engineers doing work of a kind they actually do. Its conflicts are as structural as they are obvious — Google authors, Google engineers, Google’s own AI features, one season — and it is candid about being a lab study. What it supplies is a bound on how far METR’s finding travels, and a worked example of the same gap this case is about, appearing this time between a paper’s abstract and its own regression table.⁶ Nothing in this case’s central claims rests on either.

Finally, the causal label applies to the 2025 trial’s internal validity only. Everything about 2026 in this case is Unknown or Attributed, and the ledger below reflects that.

Material claims

Direct quotations

“Surprisingly, we find that allowing AI actually increases completion time by 19%—AI tooling slowed developers down.”

— Becker, Rush, Barnes, and Rein¹ · locator: Abstract, sentence 6

“Furthermore, we show that both experts and developers drastically overestimate the usefulness of AI on developer productivity, even after they have spent many hours using the tools.”

— Becker, Rush, Barnes, and Rein¹ · locator: Section 4, Discussion, paragraph 2

“We caution readers against overgeneralizing on the basis of our results.”

— Becker, Rush, Barnes, and Rein¹ · locator: Section 4.1, Key Caveats, “Setting-specific factors”

“if I am the dedicated maintainer of a very specialized part of the codebase, there is no way agent mode can do better than me.”

— An unnamed participating developer, quoted by the authors¹ · locator: Appendix C.1.2, High developer familiarity with repositories

“my head’s going to explode if I try to do too much the old fashioned way because it’s like trying to get across the city walking when all of a sudden I was more used to taking an Uber.”

— A developer in the late-2025 study, explaining why he withheld tasks from randomization² · locator: Selected developer quotes, third quotation

“Altogether, these issues make it challenging to interpret our central estimate, and we believe it is likely a bad proxy for the real productivity impact of AI tools on these developers.”

— Becker, Rush, Cunningham, Rein, and Mahamud² · locator: “Wider adoption of AI has made it more difficult to measure task-level productivity”, final paragraph

“our study in early 2025 found that people overestimated AI’s effect on their time spent on tasks by 40 percentage points on average.”

— Joel Becker⁴ · locator: Summary, paragraph beginning “Importantly, survey results are not necessarily grounded in reality”

“We compute confidence intervals accounting for the number of developers by using clustered standard errors (not reported in the released paper, but forthcoming).”

— METR, answering the anticipated objection that sixteen developers is too few⁵ · locator: Anticipated objections, “You only had 16 developers”

“However, our confidence intervals are large, and as a result the estimate is not statistically significant at the p < 0.05 level (estimate of experimental condition on best-fit Model 2, β1 = −0.24; 95%CI = [−0.51, 0.03], p = 0.086, NS).”

— Paradis and colleagues, in the results section reporting the adjusted model that the abstract’s “about 21%” is derived from⁶ · locator: Results, RQ2, page 8

Revision notes

• 2026-09-13 — Source review. The gap this case named — a chain that does not run through METR — is still open, and the review recorded why: a search of arXiv and the wider literature in September 2026 turned up no independent re-analysis of either released dataset and no replication of the task-level design. What it did turn up is a second contrast, added as a fifth source and a fourth source family: a randomized trial of 96 full-time Google engineers on a single enterprise-grade task, finding the AI arm faster, read in full including its regression table. It is a far better contrast than the GitHub Copilot trial already cited, because its design sits closer to METR’s, and it bounds how far METR’s finding travels rather than testing it. Two things inside it are recorded as discrepancies rather than smoothed: its headline “about 21%” comes from the adjusted model its own results call not significant at p < 0.05, while the significant result is the unadjusted t-test with no percentage attached; and the abstract that calls the study “enterprise-based” in its title calls it a “lab study” in its text. Three claims and one quotation added, one “does not demonstrate” bullet split in two. Grade unchanged at B: a contrast is not a corroboration, and nothing here tests METR’s estimate.

• 2026-09-13 — Initial publication at Grade B. First case in the library labelled causal, and the label is scoped in the causal assessment to the 2025 trial’s internal validity rather than to any general claim about AI and developer productivity. Held at B rather than A despite an exceptional design: all three data collections are METR’s own, so the three source families are not three independent chains, and the case says so rather than letting the count stand. Two discrepancies recorded — METR describing its own 19% finding as 20% in February 2026, and the 21-versus-20 factor count between the v2 abstract and the blog summary. The confidence interval on the headline estimate is cited to the February 2026 post because the paper states it only graphically. peng-copilot is cited from its abstract alone, which its source note states, and carries no claim in this case.