“The 19% slowdown observed among experienced developers is not an indictment of AI as a whole, but a reflection of the real-world friction of integrating probabilistic suggestions into deterministic workflows,” Gogia explained, emphasizing that measurement should include “downstream rework, code churn, and peer review cycles—not just time-to-code.”
Broader industry evidence
The METR findings align with concerning trends identified in Google’s 2024 DevOps Research and Assessment (DORA) report, based on responses from over 39,000 professionals. While 75% of developers reported feeling more productive with AI tools, the data tells a different story: every 25% increase in AI adoption showed a 1.5% dip in delivery speed and a 7.2% drop in system stability. Additionally, 39% of respondents reported having little or no trust in AI-generated code.
These results contradict earlier optimistic studies. Research from MIT, Princeton, and the University of Pennsylvania, analyzing data from over 4,800 developers at Microsoft, Accenture, and another Fortune 100 company, found that developers using GitHub Copilot completed 26% more tasks on average. A separate controlled experiment found developers completed coding tasks 55.8% faster with GitHub Copilot. However, these studies typically used simpler, more isolated tasks compared to the complex, real-world scenarios examined in the METR research.
Read the full article here
