﻿# Coding Speed Up 74%, Shipping Up 14%: When Human Review and AI Review Plugins Both Hit a Wall, How Teams Break Through

Published: 2026-11-20 · Language: en

Original: https://yage.ai/share/ai-code-review-verification-en-20261120.html

Publisher: Superlinear Academy · 100% AI generated

[Article index](https://yage.ai/share/llms.txt)

[Other language](https://yage.ai/share/ai-code-review-verification-20261120.html)

Over the past two years, engineering teams using coding agents have shared an awkward feeling: individual engineers are writing code and opening pull requests noticeably faster, yet the pace at which the team actually ships completed features to production has barely moved.

On November 14, 2026, Harvard Business School and Economics PhD candidates Fiona Chen and James Stratton updated a [working paper](https://fion.ac/jellyfish.pdf) that backs up that intuition with cross-industry data. Partnering with engineering analytics platform Jellyfish (with enterprise client consent) and LinkedIn data, they tracked 655 firms and roughly 720,000 workers (nearly 80% of them software engineers) from January 2021 to August 2026, compiling 445 million engineering activity records that connect commits, pull requests (PRs), and shipped task tickets end to end.

Comparing early and late adopters reveals a sharp divergence: early code-completion assistants (such as GitHub Copilot) produced almost no lift in final delivery, while semi-autonomous coding agents (such as Cursor Agent and Claude Code) raised engineers’ hourly coding productivity by an estimated 73.7% and commits by 36.0% within 12 months of adoption. Yet completed, deployed tickets rose by only 13.7%—a modest gain hovering near the edge of statistical significance. The researchers used an LLM to score ticket complexity, ruling out the explanation that tasks grew larger, and checked post-release bug reports, finding that software quality did not deteriorate.

Teams are writing far more code, yet shipped functionality is not scaling proportionally. The bottleneck sits in code review, and the two most obvious responses teams reach for have already hit their ceiling in this dataset.

## Why Asking Everyone to Spend More Time Reviewing Has Hit Its Limit

Faced with a growing backlog of PRs, an engineering manager’s first instinct is straightforward: spend less time writing code and shift more people into reviewing it. The Harvard data shows that these 655 companies have already done exactly that—and pushed it to the limit. Within 12 months of adopting agents, firms cut engineering time spent on coding by over 20% and increased time spent on review by over 40%. Review now takes up 49.2% of combined coding-and-review hours. Because shifting people into review simultaneously shrinks the pool of people writing code, the researchers’ model estimates that once review crosses 49.4%, writing and reviewing reach equilibrium—shifting any further leaves upstream code supply under-utilized. Continuing to reallocate internal hours would lift final ticket output from 13.7% to at most 14.1%. There is nothing left to squeeze from internal reallocation.

Internal reallocation stalls because reviewers are not interchangeable. In actual company records, 82.8% of code reviews are performed by veterans who have previously modified that repository; under random assignment, a reviewer would land on a familiar repo only 17.2% of the time. Seniority creates a similar barrier: 75.7% of code written by senior engineers is reviewed by another senior engineer.

Real-world incidents often push teams in the opposite direction. One engineer interviewed in the paper noted that after AI-generated commits caused several production outages, the team introduced a new rule requiring senior engineer approval for all AI-written code. Scarce senior reviewers grew even more overloaded, and the more complex the codebase, the worse the jam: in the top third of firms by repository count, PR merge times jumped 69.7%.

Across the full sample, adopting coding agents lengthened total PR time from submission to approval by 56.1%, increased review comments per PR by 38.6%, and actually reduced hourly review efficiency by 7.0%. Production bug rates did not worsen precisely because approval cycles stretched by nearly 60% and teams shifted over 40% more effort into review, catching risks before deployment. The price was that extra upstream coding capacity piled up in the review queue.

<figure>
<img src="https://yage.ai/share/media/d2d3b6b1875f733a05932cb486e84d596f4c92b6c0824876eb5b7342a5221522.webp" alt="Transmission bottleneck funnel from the Harvard study: coding productivity rises 73.7%, passes through a review bottleneck where review hours reach 49.2% (near the 49.4% ceiling), 82.8% repository-familiarity barrier, and 56.1% longer approval time, yielding only 13.7% more shipped tickets." />
<figcaption aria-hidden="true">Transmission bottleneck funnel from the Harvard study: coding productivity rises 73.7%, passes through a review bottleneck where review hours reach 49.2% (near the 49.4% ceiling), 82.8% repository-familiarity barrier, and 56.1% longer approval time, yielding only 13.7% more shipped tickets.</figcaption>
</figure>

## If People Cannot Be Shifted Further, Why Haven’t AI Code Review Plugins Solved It?

When human bandwidth runs dry, the industry’s second instinct is to use AI to review AI—plugging tools like CodeRabbit or Copilot Reviewer into the repository.

The Harvard paper scanned GitHub for more than 70 AI bot accounts and uncovered a sobering gap: by July 2026, 79.1% of sampled firms had already adopted AI-assisted review tools. Yet when code actually gets merged, only 23.3% of review comments come from AI, and only 10.8% of PRs receive even a single AI comment. Humans still guard almost every merge.

Why do companies pay for AI review tools yet remain reluctant to hand over approval? Because the inspection method itself has not changed. Whether a senior engineer reads a diff on screen or an AI plugin scans the diff and leaves comments, both are doing static reading. Static reading has two blind spots: code can look spotless on paper while breaking runtime business state, and when the model writing the code and the model reviewing it share the same blind spots, edge cases missed during generation get missed again during static review.

This aligns with what I wrote in October 2026 on [what is left of code review](https://yage.ai/share/ai-code-review-attention-bottleneck-20261019.html): neither brute-force human reading nor generic AI nitpicking can rescue a high-throughput engineering pipeline. The Harvard paper’s long-run simulations issue the same warning: if review and verification productivity stays flat, even exponential gains in code generation will eventually plateau once review hits the senior-engineer ceiling.

## From Reading Code to Running Software: Why Unit Tests Fall Short and Three Boundary Tests Take Over

In the same week the Harvard paper went viral, Anthropic’s Claude Code engineer Addy Osmani published [how the Claude Code team thinks about verification](https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/) on November 16, 2026. The team self-reports that roughly 90% of its code is written and reviewed by Claude. Their core shift moves the checkpoint forward: instead of waiting for humans or AI to statically read a PR after submission, they require the agent to produce runtime evidence before committing. Without a verification loop, they argue, an engineer can supervise only one or two Claude instances at a time; with one, they can run dozens or hundreds in parallel.

At first glance, asking AI to write code and run tests might sound like a return to classic Test-Driven Development (TDD) with walls of unit tests. As I argued in [Why TDD Is Not the Answer in the AI Era](https://yage.ai/share/tdd-not-ai-native-20260508.html), traditional unit tests lock onto internal function paths. When turning tests green becomes the sole objective, AI behaves like a test-taker gaming an exam, finding shortcuts that bypass real business logic; having the same model write both implementation and unit tests also reproduces identical blind spots.

Verification that actually constrains agents must pull back from internal code paths to system boundaries—focusing strictly on end outcomes and invariants that cannot break. The three test categories the Claude Code team invests in sit squarely on that boundary.

First are end-to-end (E2E) tests paired with runnable verification instructions inside the repo. Unit tests can all pass while a core feature is broken; E2E tests are much harder to fool because they exercise real user flows and either pass completely or return screenshots and the exact failing step. The team maintains dedicated `verify` skill files close to the application code specifying how to boot simulators, drive browsers, or run local services. Before committing, Claude runs the change like a QA engineer; if it fails, Claude fixes it locally and escalates to a human only after 3 failed rounds. That keeps the most exhausting first-pass rework inside the local sandbox before any PR is opened.

Second are property tests, built to break AI’s shared blind spots. Ordinary tests check a handful of examples a human thought of. Property tests declare rules that must never break—such as an account balance never going negative, or compressing and decompressing data always returning the exact original input—and then use libraries like `Hypothesis` or `fast-check` to bombard the code with thousands of randomly generated adversarial inputs. Because those inputs are not invented by the LLM, they reliably expose edge cases missed by both the coding agent and its unit tests.

Third are differential tests, ideal for large-scale refactors and migrations. Rather than having engineers compare old and new implementations line by line, the team treats the legacy system as the oracle, feeding random inputs or sampled production traffic into both systems simultaneously, normalizing irrelevant formatting differences like whitespace, and alerting whenever outputs diverge.

Many engineers immediately ask about the bill: doesn’t running E2E, property, and differential tests on every change require Anthropic’s internal token budget? Osmani addresses that economics directly in the post. Having an agent write tests, benchmarks, and specs is a one-time token cost. Once written as deterministic software, running those checks repeatedly before every commit consumes virtually no additional tokens. Paying a one-time LLM cost to buy out hundreds of future human review cycles works for ordinary teams too.

<figure>
<img src="https://yage.ai/share/media/99acef0dcde565b4dd41ee3b4139d41b8ec7d86d53f2363824d5907f0c5c88fe.webp" alt="Comparison of static post-commit review versus pre-commit boundary verification: on the left, senior engineers and AI plugins statically read diffs, causing 56.1% longer wait times and shared blind spots; on the right, local verify runs E2E, property, and differential tests with 3-round self-repair, base-branch read-only rules, and diff-triggered checklists before involving humans." />
<figcaption aria-hidden="true">Comparison of static post-commit review versus pre-commit boundary verification: on the left, senior engineers and AI plugins statically read diffs, causing 56.1% longer wait times and shared blind spots; on the right, local verify runs E2E, property, and differential tests with 3-round self-repair, base-branch read-only rules, and diff-triggered checklists before involving humans.</figcaption>
</figure>

## Once Verification Is Delegated, How Do You Stop AI From Editing the Rules—and Capture Veteran Judgment?

Handing the verification loop to an agent immediately introduces a practical risk: if the agent fails three times locally, what stops it from deleting the failing test, adding a sleep to mask a race condition, or loosening the review rules inside the same PR?

The Claude Code team enforces a strict rule against gaming the referee: a change under review can never modify the rules judging it. Their review system always reads `REVIEW.md` from the base or default branch, never from the PR branch itself. Approvals are bound strictly to a specific commit, so pushing new code invalidates any prior approval. For flaky tests, Claude is forbidden from skipping them, adding delays, or retrying until a lucky pass; if it cannot find the root cause, it must openly delete the test and document why in the commit for human inspection rather than leaving a false green signal behind.

The other hurdle is overcoming the 82.8% veteran-familiarity barrier documented in the Harvard paper. Veterans are hard to replace because they carry years of past outages in their heads. The Claude Code team encodes those lessons into a 39-item reliability checklist, without dumping all 39 items into context every time. Each rule carries a trigger readable from the diff—for instance, checking whether old session transcripts still load safely only when session storage or message types change. A typical PR triggers 3 to 10 items, and whenever a check pulls in a human reviewer, it explains why so the reviewer does not have to guess across hundreds of lines.

To keep rules from bloating into bureaucracy, the team sets a high bar for additions. Patterns extracted from human review comments and reverts must appear in at least 2 PRs, be identifiable from a diff, survive attempts by 3 agents to disprove them, and finally pass human PR approval.

To roll out these automated gates without grinding development to a halt, the team pairs them with three safeguards. First, every commit is stamped with how it was verified—even “none”—so unverified changes remain auditable. Second, new gates start in observe mode, logging what would have been blocked to weed out false positives before switching to enforce mode. Third, even in enforce mode, if a session is blocked twice in a row, the third attempt is allowed through with a written explanation so the agent never deadlocks.

Both sources come with clear boundaries worth keeping in mind. The Harvard paper is a working paper not yet peer-reviewed, and Jellyfish’s client base skews toward established mid-to-large enterprises averaging nearly 20 years old and over a thousand employees; small startups may face less friction. Likewise, the Claude Code team’s 90% code share and 39-item checklist are self-reported without independent audit, and differential testing requires an existing legacy system as an oracle, which greenfield projects do not have.

Even with those caveats, both records point in the same direction: once code generation becomes cheap, squeezing more review hours out of senior engineers, bolting on static AI diff-review plugins, or falling back to function-level unit tests will not break the delivery ceiling. What separates engineering teams next is turning veteran judgment into a boundary verification system—one that runs automatically before commit, stops burning extra tokens on repeat runs, and keeps its rules locked safely on the main branch.
