Next to Anthropic’s continuous integration pipeline runs an unassuming little service dedicated to tracking test history for automated builds and selecting which test cases to run on each code commit. Over the past six months, the workload on this service surged 25-fold. Engineers applied three successive patches, with each buying a shorter period of stability than the last. Ultimately, the team assigned a single engineer who spent three weeks rebuilding it from scratch—an effort the author estimated would have taken an entire quarter just a year earlier. The traditional rhythm of patching first and rewriting later no longer pays off once the structure of costs shifts.
All of this comes from a retrospective: on September 14, 2026, Anthropic engineer Sachin Malhotra published a retrospective chronicling how this bookkeeping service began buckling under load, went through successive patches, and was ultimately rewritten. At first glance, I thought this was just another piece reiterating the familiar talking points about AI making CI busier and verification more critical. Only as I read on did it become clear that the real protagonist was not the CI pipeline itself. What broke down was this auxiliary bookkeeping service.
The service consists of two components. One handles bookkeeping, recording the results of every CI run; the other handles lookups, scanning historical execution records whenever a developer opens a PR to decide which tests to execute. The selection process follows deterministic rules based primarily on two pieces of information: how each test performed historically, and how closely coupled it is to the modified package. The team did not need to run the entire test suite from start to finish on every PR. Similar selection approaches are not uncommon across the industry—for example, Meta deployed a similar selection mechanism back in 2019. This mechanism relied neither on code coverage nor on machine learning classification models.
The legacy design crammed both components into a single singleton process. Ordering the test history sequentially required a single writer, a constraint that completely ruled out horizontal scaling by adding more machines. Once task throughput climbed to multiple requests per second, the bookkeeping component could not keep up; falling behind by just twenty minutes would create a backlog of tens of thousands of unrecorded test results. The CI pipeline itself never halted, and untested code never leaked into production—the trouble stemmed entirely from the ledger falling behind.
Making selections with a stale ledger wasted compute on both ends. The system would select tests already known to be flaky or failure-prone, squandering compute resources. Newly written tests failed to enter the execution pool, leaving latent regressions undetected. Broken merges took longer to surface, sending multiple engineers down fruitless debugging rabbit holes. While human engineers could triage and dismiss irrelevant alerts, coding agents relying on test loops for self-correction were left blindly retrying against stale error reports, making the disruptions of an outdated ledger especially acute.
The pressure came from several sources. The total test count grew 10-fold, while nominal engineering headcount rose only marginally. Claude tended to submit smaller, more granular PRs, driving up run frequencies. Automated workloads raised the baseline activity during nights and weekends, while daytime retained sharp burst peaks from concentrated human approvals. By May 2026, over 80% of lines of code merged to production were attributable to Claude—though a footnote noted gaps in the attribution pipeline, with the unattributed portion containing non-human artifacts such as automatically generated code. Yet in the face of frequent alerts, the team did not immediately put a rewrite on the agenda.
In October 2025, the bookkeeping service began to strain, triggering alerts for two consecutive days. At the time, no one actively owned this piece of infrastructure, and the team responsible for CI had higher-priority commitments. Engineers took the standard remediation: they doubled the CPU cores on the host machine, buying roughly 70 days of stable operation.
By February 2026, hardware resources were exhausted once again. Using code generated with Claude’s assistance, the team sharded the workload by package, converting what had been a repository-wide single-writer architecture into parallel, per-package writes. This patch bought roughly 29 days of breathing room.
In March 2026, memory pressure erupted across the board, with the service process hitting its memory ceiling nearly every afternoon. Investigations uncovered only four bugs, switching memory allocators had no effect, and the team was hesitant to run live memory profiling on a heavily loaded singleton. The team had to fall back on scheduled daily restarts, shrinking the safe operating window to under a single day. Daily restarts caused queues to snowball, with ledger lag exceeding an hour occurring repeatedly and leaving massive volumes of test data unrecorded in time.
The operating lifespans secured by the three successive patches were 70 days, 29 days, and less than a day—with each intervention yielding a drastically diminished return. This is the arithmetic reality of exponential growth: so long as input growth does not slow, any stopgap measure that expands capacity by a fixed multiple will inevitably buy a shrinking number of stable weeks.
The pressure on the input side came from a dramatic surge in code output. Anthropic’s report on recursive self-improvement noted that by Q2 2026, daily lines of code merged per typical engineer reached 8 times the 2024 level. A footnote on that page clarified that lines of code measure volume rather than quality, almost certainly overstating genuine productivity gains. Yet even accounting for the gap between quantity and quality, the accelerated influx of code was undeniable. Prior to the final cutover, backlogs appeared on monitoring dashboards on most days, with peaks of unprocessed events climbing week over week. Daily restarts could no longer hold back the surging event stream, ledger delays compounded continuously, and patching had reached the end of the road.
Faced with failing daily restarts, the engineering team finally resolved to rewrite the service. A single engineer completed the overhaul in just three weeks; by comparison, the author estimated that a similar undertaking a year earlier would have taken nearly a quarter. It is worth emphasizing that this represents the author’s personal counterfactual estimate, with no parallel control group established.
The core architectural move in the new design was decoupling state. The team separated recording run results from maintaining historical records: any worker node could accept test results, append them to an external journal, and immediately release memory upon completion. The processing nodes thus became stateless, allowing the team to scale out machine instances horizontally during traffic spikes. A separate lightweight process aggregated the external journal every few seconds, compiling it into a historical ledger optimized for rapid querying by the test selection component.
This rewrite showcased a dramatic drop in technical barrier. AI tools helped author the sharding logic and identify runtime parameters, while the storage capacity and worker node counts needed for the new design were largely derived autonomously by Claude. The cost of starting over from scratch had dropped noticeably.
Yet even as engineering costs fell, team habits remained entrenched. The author had opened a long-running chat session where Claude monitored queue backlogs and fired alerts when backlogs climbed. Over several months, Claude repeatedly recommended a clean rewrite, yet the team opted to patch the system time and again. From the initial October 2025 alerts to the launch of the new system, organizational inertia dragged the effort out for nearly half a year.
When infrastructure triggers relentless alerts, should an engineering team keep patching or halt current work to rewrite from scratch? Teams constantly weigh this dilemma. Historically, engineers routinely deferred rewrites because a rewrite demanded months of effort, whereas upsizing servers or grafting on a few conditional branches was far cheaper. Today, however, the cost of a rewrite has dropped from a full quarter to three weeks, while the time bought by patching has collapsed from 70 days to less than one. The two prices are moving in opposite directions.
| Cost of Rewriting | Hidden Cost of Continued Patching |
|---|---|
| Historically required a full quarter of engineering effort; now compressed to a single engineer delivering in three weeks. AI tools took over coding and parameter tuning, noticeably lowering the execution barrier. | Previously, a single scale-up bought over two months; now the effective weeks bought by each patch continuously shrink. Frequent alerts and restarts drive maintenance overhead higher with every round. |
| With code generation and automated reasoning, the direct investment required for a clean rebuild is falling steadily, allowing a one-time overhaul to yield lasting dividends. | Against exponentially swelling commit volumes, fixed-multiple stopgap fixes rapidly fail. Backlogged data risks disrupt engineering cadence, and an eventual overhaul remains unavoidable. |
The crossover point between the two cost curves has shifted to the left. As the cost of a rewrite trends downward and the hidden cost of maintaining the legacy system trends upward, their intersection marks the inflection point where rewriting becomes more cost-effective than patching. That inflection point now arrives significantly earlier. Rewriting can no longer wait until a service is on the verge of total collapse.
The author readily acknowledged that tactics like upgrading machine sizes, parallel partitioning, and scheduled restarts are nothing novel. The crucial insight is that two trends collided: the safety margin bought by patching shrank dramatically, while the investment required for a rewrite fell just as steeply.
This directly leads to two new decisions. First, the threshold for deferring a rewrite must be lowered: the moment the shelf-life of successive patches begins degrading at an accelerating rate, teams should initiate a rewrite early rather than burning energy on inefficient stopgaps. Second, capacity planning for initial architectures must be updated: the author recommended assuming that a system will bear a 25-fold load within two quarters, and designing the initial version with headroom for 10 to 20 times the baseline traffic.
This is the second-order dividend discussed earlier. The first-order dividend is direct: writing code is faster and requires less effort. The second-order dividend is that as cost structures change, the equilibrium point governing when to make do and when to rebuild shifts. The new architecture did not abandon conventional system design wisdom; what changed were the weights on either side of the decision scale.
Reading this retrospective, everyday engineering teams need to distinguish which lessons can be directly adopted and which practices should not be blindly copied. Much of the analysis stems from general system design principles and is not strictly tied to automated coding clusters. The applicable takeaways center on three architectural signals that do not depend on automated tooling.
The first signal is single-point vulnerabilities. A single process holding exclusive mutable state is an unambiguous scalability risk. The legacy design coupled recording and selection within the same instance; once request density surged, queue backlogs completely blocked any path to horizontal scaling. Moving state outside the process and keeping worker nodes stateless is classic engineering common sense.
The second signal is balancing inputs and outputs. Parity between inbound and outbound task volumes is the cheapest and most sensitive operational metric available. As long as incoming build tasks do not reconcile with completed records, one can conclude without complex distributed tracing that the internal ledger has fallen behind.
The third signal is the diminishing lifespan of patches. If the period of stability secured by consecutive capacity expansions continues to contract, that is an unambiguous signal to begin a rewrite. Once a system reaches this stage, continued patching merely burns engineering bandwidth in vain.
Other practices should not be copied wholesale. Small and medium teams without dense automated coding workloads do not need to replicate this entire apparatus of external journals and sharded consumers. Running tests scoped to packages and maintaining a lightweight monolith with disciplined upkeep remain entirely adequate for practical needs.
A 25-fold surge in tasks over six months was the reality for a specific team at a specific phase; it cannot be treated as standard for every team. The genuine transformation brought by new tools lies in the relative trajectory of two costs, not in any specific load figure. Stripped of a high-pressure code influx, blindly ratcheting up distributed complexity will only yield unnecessary operational burden.
In assessing this account, one must distinguish verified facts from inferences that cannot be cross-validated. This is a single case study from a single company centered on a single auxiliary service; the industry currently lacks a second set of public benchmarks.
Key data points carry clear qualifications. The 25-fold growth over six months lacks a granular breakdown, leaving the specific contributions of various factors unknown. The claimed 8-fold increase in code output explicitly admits its limitations: lines of code measure volume rather than quality, almost certainly overstating genuine efficiency gains. The statistic attributing over 80% of code contains gaps, with unattributed data conflated with non-human artifacts like automated scripts.
The engineering comparisons contain subjective elements. Shortening the overhaul from a quarter to three weeks is a counterfactual estimate that lacks a parallel control group. While the new design flattened the backlog, the author acknowledged that operating costs are higher than the legacy architecture without disclosing specific dollar figures. The retrospective provides no data on defect rates or production incidents. A flattened monitoring curve merely indicates that queue processing returned to sustainable levels; it does not prove that total verification costs fell, let alone that code quality improved.
The twin forces of acceleration and cost reduction act in tandem, and public materials cannot cleanly disentangle the two, making it invalid to multiply multiples mechanically into generalized conclusions. Listing these caveats step by step defines the boundaries of verified fact. In evaluating cases like this, I am inclined to keep my focus on verifiable price shifts, rather than extrapolating unverified inferences.