By Ravindra Dastikop
With assistance from ChatGPT
Imagine a developer asking AI to fix how an app displays dates. The patch appears in seconds. Then a reviewer asks what happens when two users in different time zones open the same record. The team adds a test, revises the patch, and reviews it again. How much time did the team save?
Two experiments measured different kinds of work
In a controlled experiment reported in 2023, developers using GitHub Copilot completed a JavaScript HTTP server task 55.8% faster than the control group, as reported by the authors. In METR’s early-2025 trial, 16 experienced open-source developers tackled 246 tasks in mature repositories they knew. When AI was allowed, tasks took an estimated 19% longer. Copilot study · METR trial
The tasks, tools, and developers differed, so the studies cannot tell us which difference drove the results. In February 2026, METR said its follow-up data gave an unreliable estimate of the size of any speedup, partly because some developers and tasks were less likely to enter an experiment that could require working without AI. METR follow-up
Count the work that moves to someone else
My view is that a team should judge AI coding assistance by how much effort it takes to deliver an accepted change at its usual quality standard. If AI saves the author twenty minutes and adds thirty minutes of review and correction, the quicker draft has shifted work to someone else. A contained task with clear tests may still yield a genuine gain. That is a reason to compare results by task type.
Try one comparison on your team
Pick a recurring task, such as adding unit tests to an existing module. Compare similar tasks with and without AI under the same review criteria. Track active author and reviewer time through acceptance, including rework. Record elapsed time separately so waiting for review does not look like hands-on effort. Treat a small pilot as a local signal and repeat it before changing team-wide expectations.
When your team says a change was faster with AI, whose time is included?

