What the AI productivity studies measured, and what to measure on your own team
The METR trial, the DORA report and the Stack Overflow survey get quoted as if they answer the same question. They don't.
Three sources get quoted whenever someone argues about whether AI makes a team faster: a METR trial, the DORA report and the Stack Overflow survey. They measure different things, and only one of them tried to time anything.
METR: a clock, on tasks, in 2025
METR’s early-2025 study is the only one of the three that measured elapsed time against a control. It recruited 16 experienced open-source developers, had them supply 246 real issues from repositories they knew well, and randomised each issue to allow or forbid AI. With AI allowed, tasks took 19% longer. The developers had expected a 24% speedup, and afterwards still believed AI had sped them up by about 20% (METR ).
The gap between the felt speedup and the measured slowdown is the useful finding. A team that asks “does it feel faster?” gets an answer that the trial shows can point the wrong way.
The authors are careful about what it doesn’t show. It doesn’t show that AI slows down most developers, or that this is how most software work looks. It says nothing about tools released since, and the sample is small. METR’s own page now says these results are out of date.
METR: why the follow-up didn’t settle it
In February 2026 METR published the follow-up: 57 developers, 143 repositories, more than 800 tasks, run from August 2025. The new developers came out at an estimated 4% speedup, with an interval of -15% to +9%. METR doesn’t treat that as the answer. It calls the data “an unreliable signal of the current productivity effect of AI tools” (METR ).
The reasons matter more than the number. Developers declined to take part because they didn’t want to work without AI, which METR says likely biases the estimate downward. In surveys, 30% to 50% of developers said they withheld tasks they didn’t want to do without AI. And time tracking broke down once developers ran several agents at once and worked on something else while waiting.
That last point applies to any team. Time per task is a weak measure when one person supervises parallel agents, because the task no longer owns the person’s attention.
METR says it is changing the design. The directions it lists include shorter, higher-paid experiments, assigned tasks instead of chosen ones, observational data on how developers use AI, and randomising by developer rather than by task. None of those is a finished replacement yet.
DORA: a survey of what amplifies what
The DORA 2025 report isn’t a stopwatch study. Its headline is that AI acts as an amplifier: “AI’s primary role is as an amplifier, magnifying an organization’s existing strengths and weaknesses” (DORA ). The greatest returns, the summary says, come from the underlying sociotechnical system and not the tools. A companion AI capabilities model describes seven capabilities that amplify the benefits.
The pages opened for this piece don’t state the respondent count or the throughput and instability figures, so none are quoted here. Read the full report for those before citing a number from it.
What the framing gives a team is a question to ask of itself: what does AI magnify here? Fast, trusted tests and small changes get faster. Slow review and flaky pipelines get a bigger queue.
Stack Overflow: what developers say, not what happens
The 2026 Developer Survey’s AI section reports usage and attitude. 80% of respondents used AI at least an hour a day, and 62% have a favourable attitude toward AI tools (n = 14,428). Trust is conditional: 48.0% trust AI when they can easily verify the output, and 12.3% don’t trust it for most work tasks (n = 14,304) (Stack Overflow ).
The AI section has no productivity figures. It’s a measure of adoption and confidence, and the verification result is the part to take away: trust follows how cheaply the output can be checked.
Questions to ask of the next headline number
The three sources answer different questions, and mixing them is how a “study shows” claim goes wrong. Before repeating a figure, check five things.
Who was measured? METR’s first trial used 16 people on codebases they had worked in for years. That says little about a new hire in an unfamiliar repository, and METR says as much. A survey of 14,000 respondents says a lot about opinion and little about delivery.
What was the unit? A task, a pull request, a deployment and a developer’s week are different things, and an agent that opens ten pull requests changes the denominator for all of them.
Who chose the work? In both METR studies developers brought their own issues. When people can hold back the tasks they’d rather hand to an agent, the comparison is no longer like for like.
Which tools, and when? The first trial used Cursor with Claude 3.5 and 3.7 Sonnet. The page itself says those results no longer reflect current impact. A figure from one model generation has a short shelf life.
Was it measured or reported? Stack Overflow’s trust numbers are reported attitudes. METR’s slowdown was recorded against a control. DORA’s findings come from the survey and the analysis behind it. Each is a valid kind of evidence, and none of them substitutes for another.
What to measure on a team
None of these three tells a team how fast it is. A reasonable reading of all of them together is that self-report is unreliable, per-task time is getting harder to measure, and the outcomes that survive are the ones the team already tracks.
Pick measures that don’t depend on how a developer feels or on how long a task took:
- Lead time from first commit to merge, and from merge to production, for the same repositories before and after a change in tooling.
- Change failure rate and time to restore, from your own incident record.
- Review turnaround and the share of pull requests reworked after review.
- Escaped defects traced to code written with agent help, which needs a label on the pull request.
Keep the comparison honest. Compare a repository to itself over a quarter or more, not one team to another, and note anything else that changed in that window. A change in tooling alongside a reorganisation tells you nothing.
The METR selection problem applies at team scale too. If developers who dislike AI are excluded from the pilot, or if people quietly reserve the tasks they find tedious for the agent, the result flatters the tool. Say in advance which work is in the sample.
One concrete step this week: add a single label to pull requests that were written mostly by an agent, and start recording merge-to-production lead time per repository. In a month there’s a baseline that doesn’t rely on anyone’s impression.