Guide
The review bottleneck: Harvard's 718-firm study shows AI coding agents write 30% more code but ship zero more features
Harvard's 718-firm study: AI coding agents write 30% more code but ship no more features, while pull-request review times grow 49%. What it means for your team, what the study cannot prove, and what to measure instead.
- Published
- 12 Oct 2026
- Reading
- 8 min
- Class
- guide
half-life 90dfrom 12 Oct 2026
On 9 October 2026, Ars Technica covered a Harvard working paper that asks the question every team using coding agents now faces: does adopting AI agents produce more finished software? Fiona Chen and James Stratton's “Artificial Intelligence in the Firm: Bottlenecks in Software Production” (current version dated 4 August 2026) draws on engineering analytics from Jellyfish — roughly 300 million work events across 718 firms and 725,938 workers, from January 2021 to March 2026. The short answer: more code, no detectable change in delivered work, and a review queue that got noticeably longer.
The authors compare each firm's output before and after it adopted AI coding assistants (GitHub Copilot, Cursor) and, later, AI coding agents (detected through signals such as Claude Code co-author lines), against firms that adopted later or had not yet adopted at all. Adoption dates are set by each firm's own legal, security and budget approvals, so this is an observational comparison rather than a randomised trial — worth keeping in mind for every number below.
More code, same finished work
After a firm adopts coding agents, per-worker output rises across every measure of writing code:
- Lines of code: +30% (baseline 5,017 per worker-month).
- Commits: +20% (baseline 22.58).
- Pull requests opened: +23% (baseline 5.34).
Finished work does not follow. Jira issues resolved rose by an estimated 0.12 per worker-month against a baseline of 3.67, with a standard error of 0.17 — statistically indistinguishable from zero. Larger work items (epics) show no significant change either, and the authors checked that firms were not simply shifting to bigger tickets that a flat count would hide.
Assistants on their own produced smaller gains: about +12% lines, +9% commits and +5% pull requests, with only the commit figure statistically significant in the paper's main estimates. Employment did not move: the authors report a precise zero effect on overall employment, ruling out a decline greater than 2.9% over the window studied.
Where the extra code goes: the review queue
The mismatch piles up in code review. After agent adoption:
- Time from pull request to merge rose by 3.45 days on a 7.03-day baseline — a 49% increase on the paper's point estimate. The estimate is significant but loose (standard error 1.60 days), so read the exact figure as a direction rather than a stopwatch reading.
- The share of pull requests with changes requested nearly doubled, from about 13% to about 25%.
- Review comments per pull request rose 35%, from 1.66 to about 2.2.
- The share of workers performing reviews rose 14%, from 29% to about 33% — while the share writing code did not change significantly.
One obvious explanation, bigger pull requests, does not hold: lines added per pull request did not change significantly (a move of −68 against a baseline of 1,168, standard error 127). The clearest evidence of strain is that reviewers push back more often and write more comments.
An interviewed junior engineer quoted in the paper put it plainly: “People could make a huge number of commits quickly, but code review was still the bottleneck.” After outages caused by AI-generated commits, their team introduced a rule requiring senior approval for agent-written code — which, in their words, “created even more review bottlenecks”.
AI reviewers have not drained the queue
The natural objection is that AI can review code too. The authors checked. By March 2026, nearly 80% of firms in the sample had used AI tools in code review — CodeRabbit, Greptile, GitHub Copilot Reviewer, Cursor Bug Bot, Graphite and Bito among them — yet AI generated only 23.3% of all review comments, and just 10.8% of pull requests received even one AI comment. Having bought a review tool is not the same as review becoming automatic.
One wording point if you cite these figures: the paper measures AI code review tools, not coding agents, and the 10.8% counts pull requests that receive at least one AI comment. Ars Technica's 9 October 2026 write-up paraphrases the same numbers slightly differently.
Automated review is now where the tooling is racing to catch up. GitHub launched ReviewBench on 5 October 2026, an open benchmark built from 219 real pull requests across 187 repositories and 19 languages, measuring both how well AI reviewers find problems and how often they cry wolf. Review quality has become a competitive surface, not an afterthought.
What the study cannot tell you
- It is a working paper, not a peer-reviewed result. It is listed as a job-market paper; the version cited here is dated 4 August 2026 (first version 7 January 2026).
- It is one vendor's customer base. All 718 firms are Jellyfish clients that opted into research use. Jellyfish reviewed the manuscript only to confirm it disclosed no proprietary information, not for its scientific content.
- It is observational. The design assumes adoption timing is unrelated to how a firm's output was already trending. The authors expect any remaining confounding to push their estimates upward, which makes the flat finished-work result harder, not easier, to dismiss.
- The window closes in March 2026. Agents have improved considerably since, and by January 2026 over 95% of firms in the sample had adopted one, so the comparison is mostly between earlier and later adopters rather than adopters versus non-adopters.
- Review signals are a proxy for quality. More change requests can mean worse code, or stricter house rules after a bad incident. The paper reports no production defect rates.
Task-level evidence points the same way from a different direction: METR's randomised trial of 10 July 2025 found 16 experienced open-source developers took 19% longer on 246 real issues when allowed to use AI tools — and still believed they had been faster.
What to do about it
The paper's own advice to managers is blunt: “Many managers rely on intermediate productivity metrics, such as lines of code or commits; however, these may not accurately capture the true gains from AI adoption.” Concretely:
- Measure the pipeline, not the keystrokes. Track time to merge, change-request rate, comments per pull request, the share of the team reviewing each month, and issues resolved per person — all available from GitHub's pull request data — before and after an agent rollout.
- Budget review capacity before buying more generation. The authors' conclusion is that “realizing the benefits of AI adoption requires complementary investments in downstream review capacity, rather than simply increasing code output”. Add agents without adding reviewers and you have only moved the queue.
- Be careful with approval rules. Routing all agent-written code through a handful of senior approvers protects quality on paper and lengthens the queue in practice — exactly what the outage story above describes.
- Take automated review seriously as a complement. AI currently handles under a quarter of review comments. Teams scaling up agents should evaluate AI reviewers with the same rigour they apply to coding agents, now that benchmarks such as ReviewBench (5 October 2026) exist to compare them.
Sources
- Fiona Chen and James Stratton, “Artificial Intelligence in the Firm: Bottlenecks in Software Production” — working paper, current version 4 Aug 2026 (first version 7 Jan 2026). The primary source for every figure above.
- Ars Technica: AI coding agents generate more code, but not more software — 9 Oct 2026. The report that brought the paper to wide attention.
- BIGCHANGE.AI: AI coding agents raised code output in a firm study, while completed work lagged — 10 Oct 2026. Figures checked against the paper, with baselines and intervals.
- SXZ.io: A Harvard study of 718 firms turns AI coding agents into a code-review queue problem — October 2026. Line-by-line verification of the paper's tables plus queue arithmetic.
- METR: Early 2025 AI experienced open-source developer study — 10 Jul 2025. Randomised task-level trial pointing the same direction.
- GitHub: ReviewBench, an open benchmark for AI code review — 5 Oct 2026. 219 real pull requests across 187 repositories and 19 languages.