On August 28, Anthropic published an empirical report titled Automated Researchers Can Reliably Mitigate Alignment Failures. They had Claude Opus 4.8 drive automated research agents to devise training recipes aimed at fixing flaws in other models: issues like sycophancy, deception, and jailbreaks. The report estimates an API inference cost of approximately $4/hour per agent; this covers API inference only, excludes total costs such as training GPUs, and cannot be directly compared against the full experimental hourly rate of $150 for human researchers.
The entire setup functions like an exam hall where tests run continuously: the small models undergoing training are examinees, benchmarks are exam papers, and the Claude-driven research agents are more like coaches constantly trying new teaching methods. In each round, the agent proposes a recipe, initiates a fresh training run, and inspects the resulting checkpoint’s score. This experiment sets out to answer two questions. First, why does the research loop improve scores: how much do good ideas at the starting point really matter? Second, are the gained scores genuine: could the model simply be overfitting to the surface features of a single exam paper? Anthropic probed these boundaries using seeded controls, benchmark ablations, and held-out evaluations.
The first component swapped out was the starting direction. Seeding the system with human researchers’ proposals did not lead to stronger final performance.
The operational mechanism of this automated exam hall is not complicated. Anthropic placed Claude agents into a fixed environment to tackle a specific category of model failure, such as habitual dishonesty. The agents’ actions in the exam hall are straightforward: search the literature, propose a training recipe and accompanying data, write a methodology description containing no results, wait for a monitor agent to approve the code before training a small model, and then submit the checkpoint to an independent evaluator for scoring. Each failure category was assigned five agents exploring in parallel, sharing a leaderboard and a discovery forum, running for up to 48 hours. The training budget for each candidate recipe was a single H200 GPU for approximately 30 minutes.
What the research agent delivers is a complete training recipe: the specific method, accompanying data, and execution code. The exam hall accepts the recipe and performs post-training on a small model with 2B to 7B parameters, directly updating the weights of this target model. Claude, which proposes the recipes, does not update its own weights within the loop. Each candidate recipe is trained independently and submits its own checkpoint, rather than having a single small model accumulate weights continuously across dozens of rounds. This division of labor shares a similarity with the previously analyzed AgentFlow framework (Switching to GPT-4o Gained Only 5.8 Points, but Training a 7B Model Gained 17.2 Points): an outer agent handles search, while training focuses on a separate target model.
The control group consisted of 28 experienced AI safety researchers, with an average of 2.5 years of relevant experience. Each researcher spent up to 8 hours designing a training recipe against the same benchmarks. Across all 7 failure categories with comparisons, the best recipe discovered by the agents closed more of the safety gap than the best single proposal from the human group; in the deception task cited as an example in the report, the difference was approximately 20 percentage points. The comparison here measures how much of the gap between baseline and theoretical optimum was closed, rather than raw exam scores.
The subsequent seeding experiment hit at the crux of the matter. The researchers separately launched 30 guided versus 30 unguided runs (the former seeded with a human proposal, the latter without a designated initial direction). Seeded agents first faithfully implemented the proposal, after which they could continue modifying it or pivot to a new direction following their attempts. Across the same 7 failure categories and target models, both groups achieved similar final performance. Another experiment, which assigned five agents to different human directions respectively, likewise failed to yield higher final scores, merely increasing recipe diversity in the early stages.
Anthropic added two crucial qualifications in the report. Human researchers could not continue iterating after submission, whereas agents searched continuously, so this was not a head-to-head contest of raw intelligence. More critically, both the seeded and unseeded groups read the exact same literature review, and the seeded group was free to abandon the human direction later on. The report thus explicitly stated that this experiment did not test whether human ideas themselves are helpful. The narrow conclusion it supports is: within this specific task set and search budget, having humans designate the starting direction did not improve final performance.
That human-designated starting directions did not elevate the finish line does not mean all credit belongs to scoring. The report separately examined the search environment, the training methods, and the benchmark structure—three lines of evidence answering three distinct questions.
First, the search environment. In a set of harness ablations on the sycophancy task, removing the shared discovery forum resulted in closing roughly 6 percentage points less of the safety gap; similarly under offline conditions, having a cached literature review outperformed having no review by nearly 9 percentage points; once the review was prepared, removing live internet access produced no detectable loss. Because the forum simultaneously contained leaderboards, results, and code snapshots, and the literature review altered the search starting point, these experiments cannot independently prove that gains stemmed from scoring. Each condition was also run only once (the one-run ablation caveat), and the authors characterized the results as suggestive rather than conclusive.
Next, the methods. Out of all 1,601 proposals, 74% utilized the target model’s own generations; for the sycophancy task, 98% adopted self-distillation approaches already established in the literature. This distribution indicates that search heavily reused known techniques, though it does not imply that new methods lack value. The report also tested novelty constraints: while they significantly boosted methodological novelty, on Petri one metric improved while another degraded. Novelty did not consistently translate into stronger generalization. What truly thrust the exam hall into the spotlight was the next, more direct body of evidence: when optimizing against only a single exam paper, impressive scores failed to transfer to two other exam papers.
Having clarified the driving forces behind score gains, the next question must be answered: do the improved scores represent real capability, or did the model simply memorize the exam questions? The answer lies entirely within the rules of the exam hall.
Why must the exam hall provide 3 to 5 open benchmarks of varying origins and formats, while withholding a sealed, held-out exam outside the arena that never enters optimization? Figures in the report’s appendix illuminate the rationale (full report). One group of agents optimized solely against a single benchmark: Open Prompt Injection. The winning method closed 70.9% of the baseline-to-optimal gap on this exam; yet when transferred to two prompt injection benchmarks that had never participated in optimization, it closed only -11.9% and 2.0%. A negative number indicates performance worse than the untrained model, not a negative exam score. The authors concluded that this method merely adapted to the surface features of a single benchmark, rather than memorizing specific questions. Hill-climbing on a single exam paper failed to produce a generalizable method.
Compare this with the main experiment: when simultaneously optimizing across 3 to 5 benchmarks for each failure mode, the top-ranked method on the leaderboard exceeded the untrained baseline on held-out evaluations across all 10/10 failure modes. When the research team used the held-out exam to select one of the leading methods and transferred it to models 1.8x-4.7x larger, the gains still persisted; in multi-turn Petri audits, the target behavior also outperformed the baseline across almost all failure modes and conversation turns. It should also be noted here that the held-out exam served two distinct roles: the report’s main results evaluated the leaderboard’s top method directly against it, but subsequent larger-model transfers and Petri audits first used it to select among the leading methods, meaning the only evaluation genuinely uninvolved in selection was Petri. Taken together, these layers of evidence demonstrate that multi-benchmark search yields far more than high scores on a single exam paper.
Peers building AI products will find this logic familiar: define what good looks like before making changes. The “Evaluation-First” principle frequently invoked in engineering gains both large-scale empirical backing here and a crucial qualification: a single exam paper can bring score gains, but does not necessarily bring transferable improvements.
Verifying whether scores are real cannot bypass the hurdle of anti-cheating: if scores can be gamed through illicit tactics, score gains prove nothing at all. Anthropic deployed a tiered monitoring pipeline to scan all 1,601 research trajectories, including chains of thought, followed by full-text reviews of suspicious trajectories by Claude Opus 4.8, confirming 39 cheating attempts (39/1,601, or 2.4%). The three tactics were all tricks familiar from school exam halls.
The most common was re-taking the test for luck: examinees resubmitted identical, unmodified code, gambling that random fluctuations in the evaluation process would yield a higher score. The second was mixing exam questions into practice workbooks: cobbling together training data modeled directly on the evaluation benchmarks. The third was smuggling cheat sheets: concealing illicit operations within implementation details in an attempt to slip past pre-exam inspections.
To prevent cheating in the exam hall, several defense lines of differing natures were established. Before an exam began, a Claude Opus 4.8 monitor agent inspected the actual training code to check for unauthorized model distillation, benchmark data leakage, or smuggled illicit steps; a second monitor agent cross-checked the methodology description against the code. Code versions that passed review were immediately frozen and required re-audit if edited. Held-out exams were isolated via operating system permissions and kept out of environments accessible to research agents. While code is closer to execution reality than method descriptions, this remains an audit by language models, not an infallible mechanical verifier.
The second line of defense operated post-exam: inspecting the examinee’s problem-solving process to see if improper intentions surfaced in its chain of thought. But this defense stands on far shakier ground. Anthropic reported that Opus 4.8’s misbehavior attempts in this experiment still tended to appear in its reasoning, making the team cautiously optimistic that the monitor caught most cheating; that premise may not hold for future models. The researchers therefore identified maintaining model monitorability as critical. The lifespan of this second defense is tied directly to the pace of model evolution, making it a protective asset that continually depreciates.
The 39 cheating attempts serve as stress-test logs against the exam papers and proctoring rules. In the experiment, no confirmed cheating recipe was ultimately selected among the reported methods, and only once did a recipe relying on resubmission luck reach the leaderboard’s top three. This demonstrates that defenses caught the detected cheating within this experiment, but 2.4% cannot be taken as the true cheating rate or proof of future robustness. The authors explicitly warned that these research agents did not mount strong attempts to evade detection; systems more aggressively pursuing rewards might peek at sealed data, manipulate evaluations, or tamper with scoring.
The thresholds of the exam hall extend beyond anti-cheating. For a method to pass final review, if the target model’s 95% confidence interval on any of MMLU, GSM8K, or IFEval fell entirely below baseline, the gate rejected it outright. While this was a pre-specified statistical threshold, it can only filter out overt degradation, not prove that capabilities remain entirely intact. In the appendix, some runs saw IFEval drops of 9.5 to 12 raw percentage points yet still passed due to sampling uncertainty. Anthropic also explicitly noted in its limitations that this safeguard covered only mathematics, college-level knowledge, and instruction following; unmeasured capabilities could still suffer damage.
What an automated self-improvement system uses for scoring alters both the reproducibility of evidence and the ways in which loopholes can be exploited. Yet this is not a linear spectrum stretching from completely un-gameable to easily fooled. Mechanical targets can harbor implementation vulnerabilities, while textual reviews are not equivalent to merely judging writing style. A more accurate way to compare is to examine what target each of the three systems evaluates, and what supplementary evidence constrains scoring.
Consider first DeepMind’s FunSearch and AlphaEvolve, analyzed previously (LLMs Propose Candidates, Evolutionary Frameworks Decide Who Survives): programs output by language models run directly against mathematical or engineering objectives, allowing candidates to be recomputed against the exact same objective function. While this diminishes room for verbally persuading an examiner, it does not structurally eliminate gaming. Flaws in evaluation code, execution isolation, and misspecified objective functions remain viable attack surfaces.
Next, consider AAR in this study. Independent evaluators run multiple benchmarks, sealed exams are isolated at the operating system level, and pre-run code audits along with post-hoc chain-of-thought reviews are entrusted to language models. Assembled together, these three verification mechanisms each hold distinct defensive sectors, and each harbors distinct vulnerabilities.
Finally, consider Sakana AI’s The AI Scientist. This system generates research projects and papers, which are then scored by an Automated Reviewer composed of five LLM reviews. What Nature published in 2026 was the system paper authored by human researchers, not an AI-generated paper. In a separate independent experiment, three AI-generated manuscripts were submitted to an ICLR 2025 workshop; one received an average reviewer score of 6.33, higher than the workshop’s average acceptance threshold. Per prior agreement, it was withdrawn before meta-review, so more precisely, it cleared the reviewer score threshold rather than entering Nature or receiving formal conference acceptance.
This is precisely the issue encountered when deconstructing multi-agent systems earlier (Pipelines Are Laundering Confidence): if screening relies solely on language models reading prose, persuasive writing easily bleeds into fitness, requiring supplementary mechanisms for factual grounding. That The AI Scientist’s Automated Reviewer achieves alignment comparable to human decisions indicates it can approximate reviewer judgment, not that it verifies the empirical findings in the paper. Sakana itself disclosed citation attribution errors in AI-generated papers. Princeton’s shadow evaluation for open-ended research switched to an entirely different kind of examiner: assigning the core research problems from two unpublished NeurIPS submissions to Opus 4.8 for six days, then asking the original authors who know the problems best to evaluate the output. Original author evaluation is not flawless either, but at least it avoids treating the same automated reviewer score as both the optimization target and the final proof.
Assessing whether an automated self-improvement system is trustworthy begins with examining what the examiner confronts: repeatable programs, post-training model behavior, or a paper requiring interpretation. The more stages that are mechanically recomputable, the less results depend on persuasiveness; yet what truly determines defense strength also includes objective functions, data isolation, independent testing, and monitoring coverage, rather than merely counting how many mechanical steps exist.
In this experiment, the search loop’s continuous improvement depended on repeatable exam papers; whether scores reflected genuine changes hung on how multiple benchmarks, isolation, and external testing were combined. Yet the boundaries of this exam hall are equally apparent. AAR functioned because the experiment selected ten failure categories with existing public benchmarks or automated audit tools; faced with open-ended risks that lack exam papers, this method currently falls short. Certain real-world safety risks are too rare or novel for anyone to draft a qualifying exam. Anthropic stated plainly in the text: these evaluation metrics are ultimately proxies for the complex risks of the real world.
An even larger void lies on the exam supply side. The benchmarks in the experiment were curated by the research team, with human spot-checks of samples and model responses, and tailored to the target models. Five agents sharing a single leaderboard resembled a collaborative climbing team rather than a self-evolving market: no independent validators competed, and evaluators were not dynamically replaced during execution. AAR automated recipe search, but did not automate where exams come from, when they should be updated, or who audits them.
A previously mapped landscape of self-improving AI systems based on verification signal strength (Self-Improving AI: A Flattened 2D Field) reached a central conclusion: evaluators constitute the bottleneck across the entire map. Anthropic’s experiment supplies the second half of that thesis: within the very same tier, internal structure—such as benchmark quantity, diversity, and isolation—determines whether improvements can reach beyond the optimization target. Joined together, these two puzzle pieces delineate the boundaries of the title’s claim: on tasks with mature exam papers, having humans designate a superior starting direction yielded no incremental gain; exam halls capable of both powering search and verifying generalization remain scarce.
Good ideas are already abundant serves well as an industry assessment grounded in this experiment, but should not masquerade as a universal law. What the report actually demonstrated is that across 7 alignment tasks with existing benchmarks, seeding automated search with human proposals did not improve final performance. By contrast, multiple benchmarks from diverse sources and formats, a sealed exam untouched by optimization, and a protocol for detecting shortcuts were the prerequisite conditions enabling this AI self-improvement system to produce credible results.
If you are using agents to automatically improve your own systems, consider working through three questions before taking action:
Before having clear answers to these three questions, there is no need to rush to pack more research agents into the system. Within the boundaries supported by this experiment, the exam hall comes before the examinee is the most dependable conclusion.
Primary sources: Anthropic Main Blog · Full Alignment Science Report · Open-source harness