Figure 1: Layers of self-improvement divided along artifact
persistence, code search, weight updates, and research
autonomy.The most alluring aspect of AI self-improvement is the prospect of turning model development into an accelerating loop. Models participate in generating data, designing rewards, writing experiments, and running training, then feed the results back into the next round. With each turn, the system can try more variants, get feedback faster, and possibly spend less.
Still, AI participating in R&D is a step removed from AI itself driving the progress of the next generation of models. Today’s agents can already write code and run experiments, but what to research, how to score, how much compute to allocate, and whether to accept the results—these decisions typically remain with humans. Self-improvement in the strict sense demands more: the system must not only improve the core capabilities of the next-generation model, but also ensure that these improvements are retained in subsequent iterations. Existing public evidence primarily supports the former capability—R&D automation.
Over the past several years, quite a few public prototypes have emerged, each connecting a different segment. Eureka has large models write reward functions and then train inner policies via reinforcement learning, but the outer model that writes the rewards does not update with the results. AI Scientist connects the pipeline from ideas through experiments to papers, yet the agent responsible for research is itself static. AlphaEvolve can search for and optimize training kernels, but Gemini itself does not learn in the loop. DGM, which allows agents to modify the codebase, likewise freezes the base model. PostTrainBench turns automated post-training into a benchmark, and the results show agents still significantly lag behind official instruction-tuning baselines.
Dan Austin’s ai-trains-ai v0.1, released on July 14, 2026, places these scattered pieces into a single public project. It is not the first AI-trains-AI experiment, and outside researchers cannot fully replicate the entire process. But within one repository, we can simultaneously see how the outer policy learns, how the inner model is trained, whether the results transfer, what went wrong in operation, how much cost the author reports, and whether the evaluator measured correctly. This project therefore lends itself well to answering a more specific question: where exactly has so-called self-improvement already formed feedback loops, and where does it still depend on the tracks laid by humans.
Figure 1: Layers of self-improvement divided along artifact
persistence, code search, weight updates, and research
autonomy.
The project code is publicly available on GitHub, and the trained outer LoRA weights are hosted on Hugging Face. The creator Dan Austin describes himself on his public profile as a Principal Software Engineer at Microsoft and is the only visible human maintainer of the repository. The README also notes that Fable 5 wrote every line of code in the project. Dan had previously worked on a calculator reinforcement learning task, the TerminalBench infrastructure, and a multi-agent evaluation framework—efforts that eventually converged into this project.
The outer agent uses Qwen3.6. The task it receives is something like “train a small model to handle multi-step queries,” but it does not directly modify model weights. Instead, it must first submit a complete post-training plan, consisting of two types of files: one that defines the training data, the tools the model can invoke, and the reward rules used during training; the other that decides which base model to start from, along with hyperparameters such as the learning rate, number of training steps, and batch size.
The base model here is the starting point before inner training begins. The project only allows the agent to choose between Qwen3-0.6B and Qwen3-1.7B. Once selected, the GPU begins post-training from that raw checkpoint. Therefore, “choosing 1.7B” is not picking a winner between two training results—it is deciding, before training starts, which model to feed into the training pipeline.
The actual execution of inner training is handled by prime-rl. It is
a framework for running reinforcement learning post-training, and the
project provides default training configurations. The outer agent can
also modify the sampling temperature, optimizer, algorithm variant,
learning rate schedule, and loss in the [prime_rl]
configuration table within its submission. In other words, prime-rl is
not another model being selected—it is the tool responsible for actually
running the training plan.
After passing pre-validation, the submission enters the GPU queue and can only be trained once. The outer agent does not see the training results and has no opportunity to revise the plan based on those results. Once training completes, a hidden evaluation program tests the small model on a set of undisclosed questions, recording its absolute post-training performance and how much it improved over the best untrained baseline. These two results are then combined with validation efficiency and training speed to form the outer reward, which is used to update the agent’s LoRA.
So there are two layers of learning in this experiment. The inner small model undergoes post-training according to the plan written by the agent; the outer agent gradually changes what plans it will write in the future, based on the reward each plan ultimately receives. Humans still pre-specify the task type, evaluation method, reward formula, model options, toolchain, and budget. It is more like an agent that is learning how to design post-training experiments—not yet a scientist capable of independently deciding research directions.
Figure 2: The feedback loop formed by the outer agent generating
jobs and real inner training, nested together.
The entire outer training ran for 54 steps. The author reports that the composite reward reached its peak value of 0.627 at step 50 and stood at 0.549 at the end of step 54. This reward is not a loss, nor is it a percentage of research capability. It is a training signal designed by the author for this environment: validation efficiency accounts for 0.35, inner training results for 0.60, and training speed for 0.05. Its absolute value cannot be compared with other projects; its purpose is simply to let GRPO know which training plans should appear more frequently in the future.
Separating the composite reward reveals two stages in the learning process. The early changes are mainly about experiments running through more often. Invalid submissions decrease, and jobs that pass validation and complete GPU training increase, but the post-trained small models do not noticeably improve. In other words, the outer agent first learns how to submit an executable training plan.
Changes related to model effectiveness only begin to appear in the latter half. There are two metrics here that are easy to confuse. Post-training score is the inner small model’s performance on hidden questions after completing post-training. Job quality score combines this post-training score with “how much it improved over the best untrained model.” The best untrained model here refers to whichever of the 0.6B and 1.7B models scored higher on the hidden evaluation before receiving this round of post-training. According to the author’s batch-aggregation methodology, job quality score rose from roughly 0.30 to 0.41, and post-training score also moved from a noise level around 0.04 into the 0.22 to 0.48 range. This is the quantitative signal in the experiment that comes closer to “the training plan is beginning to produce results.”
This set of numbers still requires cautious interpretation. The project has not released the complete artifacts from each iteration, so we cannot independently reconstruct why each stage improved. The post-training score also comes only from the project’s own hidden synthetic tasks and does not represent real-world general capabilities.
The change in model selection helps us see what the outer policy
concretely changed. In the early phase, only 42% of training jobs
started from Qwen3-1.7B; in the later phase, this proportion rose to
95%. The 1.7B model has more parameters and is generally stronger than
the 0.6B, but it trains more slowly, costs more, and being larger does
not guarantee greater improvement under a limited training budget. The
agent can call get_baseline_scores to check the
pre-training starting performance of both models, so this change is not
profound: it learned to read the available signal and shift most of the
training budget toward a starting point more likely to yield a high
score. This demonstrates that the outer reward did indeed alter the
policy’s behavior, though it does not yet constitute research taste.
The proportion of jobs using [prime_rl] rose from 21% to
nearly 78%, indicating that more and more jobs were no longer fully
accepting the default training parameters, but instead actively setting
the sampling temperature, optimizer, algorithm variant, learning rate
schedule, and loss. This again confirms that the way training plans were
written changed, but without ablation experiments by the author, there
is no way to tell which specific configuration genuinely improved the
results.
Finally, let us look at the task that did not participate in outer training. The project retained a task named triage, which was tested 10 times at each of four stages, yielding average scores of 0.399, 0.438, 0.545, and 0.492. Step 34 is clearly higher than the starting point, but step 54 shows some regression. This curve only supports limited, non-monotonic transfer: some of what the outer policy learned can carry over to a new task of the same type, but the existing data is insufficient to demonstrate stable and broad generalization.
Figure 3: The trend of composite outer reward and the triage holdout
task’s test scores.
The project’s evaluator
retrospective documents a very specific bug. The old tool-usage
detector, when reading function names, looked for
.function.name, but the real runtime object only had
.name. Early unit tests and validation scripts used a
different data structure and never hit this real code path. As a result,
the tool-usage score, which carried a reward weight of 0.4, never fired
a single time in early tests.
The author writes that the reward increases seen in the old experiment actually occurred within the illusion created by this measurement error. This should not be interpreted as the agent deliberately cheating. It illustrates something else: as long as the feedback definition is flawed, reinforcement learning can still optimize a steadily rising reward curve. The evaluator is not merely a scoring tool applied after the experiment ends—it is itself part of the learning environment.
After discovering the error, the author discarded the old checkpoints, redesigned the tasks and evaluation criteria, and started training from scratch. The publicly available 54-step curve now comes from the corrected rerun.
The most interesting result of ai-trains-ai is not that
the reward reached 0.627, but that outer reinforcement learning
genuinely changed the behavior of a training agent. It first learned to
reduce invalid submissions, then gradually changed its choices of base
model and training configuration; the inner small model completed real
training, and the hidden evaluation results in turn updated the outer
policy. On the triage task, which did not participate in training, this
change even showed limited transfer.
But this conclusion only holds within the environment pre-built by the project. The tasks were chosen by humans, the reward was designed by humans, and the tools and budget were provided by humans. What the outer agent learned is how to write higher-scoring training plans under this set of rules. The existing evidence is not sufficient to show that it has acquired general research taste, much less that it can autonomously design the next generation of models.
The next time you see an AI self-improvement experiment, the first thing to ask should not be how much the reward rose, but three things:
These three questions separate proxy metrics, behavioral change, and
real outcomes. ai-trains-ai has already connected the first
two steps, and has also provided preliminary signals for the third. What
it demonstrates is not a completed self-evolving system, but a
well-bounded, partial feedback loop that can continue to be
verified.