When writing skills for AI agents, our default instinct is often to teach them knowledge: industry background, domain facts, product details—we’re tempted to cram the entire internal wiki into them. But after using them for a while, it’s easy to notice: the more background knowledge you pile on, the smarter the agent doesn’t get at doing its job, and those painstakingly prepared domain facts seem to have had virtually no effect.
A paper released in mid-August by researchers from Princeton, UCSD, Stanford, USC, and Johns Hopkins (arXiv 2608.14036) offers an explanation at the mechanistic level. The researchers compiled complete experimental traces from 8,135 agent runs and analyzed every case where a skill actually made an impact: 65.7% of the time, it worked by providing clear step-by-step guidance and checklists for actions; cases where it actually filled in missing facts for the agent accounted for only 4.5%.
By comparison, skills function much more like checklists than textbooks. This study helps clarify three things: how experiments prove that what actually makes skills work is standardizing actions rather than supplementing facts; why experience without success/failure labels backfires; and why you don’t need to over-worry about retrieval hit rates as your skill library grows.
The experimental design of this study can be understood through the lens of onboarding a new employee.
The researchers first had bare agents with zero priors independently execute a batch of complex terminal and tool-use tasks, saving full recordings of the operational process—logging everything regardless of success or failure. In the second step, the researchers organized that same batch of recordings into two distinct formats of training materials: Material A was cleaned raw execution logs (called Workflow Memory in the paper), equivalent to handing a newcomer a senior colleague’s raw run-log to cross-reference on their own; Material B was distilled by senior staff after reviewing the recordings into a one-page operating guide (a SKILL.md file), specifying what dependencies to install in the environment first, which pitfalls to avoid in advance, and what verification commands to run upon completion.
Material A and Material B came from the exact same information source, establishing strict homologous control: the total amount of experience was identical, with the only variable being its presentation format. Subsequently, three groups of agents (the baseline group with no priors, Group A with raw execution logs, and Group B with operating guides) were evaluated across hundreds of runs on the same tasks.
The evaluation results showed stark differences. In terms of final success rates, the baseline group scored 59.1%; Group A with raw execution logs saw no improvement and instead dropped to 55.9%; while Group B with operating guides rose to 61.9%. Group B outperformed Group A by 6.06 percentage points, and this 6-point advantage holds up statistically (95% confidence interval +0.76 to +11.36). With the raw source material being completely identical, this 6-percentage-point edge came from the distilled and structured format of the experience, not the quantity of experience.
Timeout failure rates tell the same story: Group A had a timeout rate of 10.6%, compared to just 1.7% for the baseline group and 4.4% for Group B. The reason is that raw logs are riddled with trial-and-error backtracking, environment debugging, and dead ends. Ingesting too much unfiltered, ineffective information not only dilutes the agent’s attention but also exhausts its time budget while repeatedly chewing through historical noise. Experience itself is not automatically an asset; raw, unrefined, and unstructured experience often turns into a liability in real-world execution.
How exactly do guides make an impact? The researchers analyzed 528 same-task comparisons side-by-side (spanning SkillsBench, Terminal-Bench 2.0, and Terminal-Bench-Pro, totaling 1,584 evaluations) and attributed the effective cases: the guides that worked were almost all providing the agent with an actionable sequence of steps, execution order, checklists, tool sequences, and verification plans—which the paper calls procedural anchor, with checklists being the most intuitive component; whereas guides that worked by supplying facts the agent didn’t already know—what the paper calls knowledge injection—accounted for a negligible fraction. The actual stats are the 65.7% vs. 4.5% mentioned earlier. The vast majority of effective cases rely on steps and checks; those that work by injecting knowledge are exceedingly rare.
The distribution of failure modes further delineates the capability boundary. Environment setup and infrastructure configuration failures dropped from 5.3% in the baseline (1.7% in the raw log group) down to 0.2% in the guide group; output format mismatch errors dropped from 7.4% to 3.2%; background service management failures dropped from 2.7% to 0.8%. These errors that saw dramatic drops share a common trait: once discovered and properly written down, they can be reused long-term. Conversely, errors that guides struggle to fix barely budged: algorithmic logic errors only shifted slightly from 8.3% to 7.4%, and static verification mistakes without runtime checks only went from 12.5% to 11.7%.
A surgeon walking into an operating room doesn’t lack anatomical knowledge; the strictly executed checklist in the OR is there to prevent missed steps under high cognitive load. The same holds true for agents in complex engineering: task failures mostly stem from execution jitter and attention drift across long chains of operations. This aligns with the perspective we published this January in From Process Certainty to Result Certainty (this is our interpretation, not the paper’s wording): runtime stability comes from an execution-and-correction closed loop paired with explicit acceptance criteria; the truly essential part of a skill is the checklist-style step-by-step guidance and verification plan—the solidified, stored engineering artifact of that certainty. The value of distilling a guide lies in compressing the heavy token cost of initial exploration into a high-density, one-page operating guide, allowing subsequent tasks to reclaim most of that certainty at the context cost of just a single page.
When writing a skill, it’s worth checking every paragraph: is this an action procedure or a factual statement? Executable steps, environment checks, and verification commands travel down the 65.7% high-efficiency channel, while piles of background facts and conceptual encyclopedias are squeezed into the 4.5% low-efficiency channel.
Since distilling operating guides yields tangible gains, automation is the natural next thought: feed all of an agent’s past execution logs into an LLM for automated summarization, crystallizing them directly into a skill library. But a set of controlled sub-experiments designed in the paper yields a clear negative verdict on this approach.
In this set of controlled sub-experiments, researchers used the exact same pool of execution logs for distillation, with the sole variable being whether the summarization model could see the success/failure labels of each attempt during distillation. In the Gemini evaluation on Terminal-Bench-2, when the execution log pool used a mixed ratio of three successes and two failures out of five attempts (noted as 3s2f in the paper), the guide generated with success/failure labels allowed the model to achieve a score of 0.7462; once the success/failure labels were hidden and an indiscriminate summary was made, the score plummeted directly to 0.4000.
An all-failure log pool (all five attempts failed, noted as 0s5f in the paper) illustrates the problem even further: even when explicitly labeled during distillation that every attempt was a failure, the quality of the resulting guide was still lower than the baseline with no reference materials at all. In Codex’s tests on Terminal-Bench-2, carrying an all-failure operating guide yielded a score of 0.5161, carrying all-failure raw execution logs yielded 0.2839, while the bare baseline without any materials scored 0.5935. When historical attempts are all failures, forcibly distilling “takeaways” ends up calcifying the failed exploration paths, performing even worse in actual execution than a blank baseline.
Failed exploration traces are not without value—they define the boundaries of trial and error—but only on the prerequisite that they carry explicit negative feedback labels. If success/failure labels are stripped away, the summarization model cannot tell which actions contributed to success and which led to failure. The resulting guide will often enshrine dead ends, inefficient retries, and even erroneous debugging operations as standard practice, teaching them to subsequent agents.
This adds another layer to the training material analogy: newcomers doing the work need to constantly cross-check acceptance criteria to keep their actions from drifting, and the chronicler responsible for writing the training materials needs even more to clearly know before writing whether each attempt succeeded or failed. If the chronicler themselves knows nothing about past successes or failures, the resulting training manual becomes an actively misleading document.
For teams trying to build self-evolving agent systems, there is a clear engineering bottom line here: automatically mining and crystallizing skills from execution logs must be predicated on every execution record carrying an accurate, trustworthy outcome verdict. Automated crystallization disconnected from an outcome verification closed loop will not only fail to improve system capabilities, but will continuously inject erroneous prior biases into the system.
When building skill assets, only workflows validated through real closed loops should be solidified into guides. Committing untested exploratory drafts directly into the library is not a neutral accumulation—it continually injects noise across the entire system.
To share something from our own experience: we maintain an open workspace context infrastructure containing a skill writing guide, written at the end of March this year—nearly five months before this paper. Its two core requirements, in hindsight, align squarely with the paper’s conclusions: first, a skill must specify acceptance criteria, concrete enough that an agent with zero prior context can determine whether it is done; second, known pitfalls are only allowed to come from real failures, reworks, misjudgments, or lessons learned across multiple iterations, explicitly discouraging concocting hypothetical pitfalls just to fill space. The former corresponds to the verification plan in the paper, while the latter corresponds to the success/failure signals the paper emphasizes. Engineering intuition and over 8,000 controlled runs arriving at the exact same conclusion—that’s probably the joy of experience preceding evidence. The guide is here: grapeot/context-infrastructure’s Skill Writing Guide.
As the number of entries in a skill library grows, a common worry arises: once the library expands to hundreds of files, if an agent fails to precisely retrieve the target guide during execution, will the task simply grind to a halt? Real-world test data on retrieval and execution yields a counterintuitive answer.
In practical tests, as the candidate skill pool expanded from 5 to 100, the proportion of skills actually inspected or invoked by the agent that happened to hit the ground-truth guide dropped from 29.6% to 3.3%; yet the final success rate of downstream tasks did not drop—instead, it rose from 36.4% to 39.3%, remaining essentially steady overall.
The underlying reason is that an agent using a skill library is more like browsing a library than answering a single-choice question. In real-world terminal operations and tool invocations, many skills share underlying engineering conventions: virtual environment initialization, spawning background daemons, troubleshooting common error codes, and validating output files. Even if the retrieval system fails to pick the preset ground truth, as long as the agent extracts usable configurations, dependency commands, or invocation formats from a few related guides it browses, it can assemble the critical steps needed to solve the problem. Therefore, exact retrieval hit is not a necessary condition for task success.
Conversely, even if an agent inspects or invokes the exact matching standard operating guide, it may still ignore the steps in the guide due to attention bias or reasoning fluctuations. Exact retrieval hit is likewise not a sufficient condition for task success.
Yet this does not mean we can let skill libraries grow completely unchecked. Offline diagnostic experiments show that while an increase in total entries creates pressure, a swarm of lookalike distractor entries is by far the bigger culprit. Once distractors highly similar to the target entry are mixed into the candidate pool, the proportion of times the system ranks the correct guide first drops from 70.5% to 53.4%; in an irrelevant random candidate pool, this proportion only drops from 97.7% to 84.1%; and in a dissimilar candidate pool, it only edges down from 96.6% to 93.2%. If a library is cluttered with obsolete, overlapping, and vaguely scoped entries, it creates severe semantic confusion at the retrieval layer.
Even when a guide is found, the execution layer pays a tangible penalty: experimental logs show that patterns of misusing or ignoring guides accounted for 10.0% of failures in the skill-equipped experimental group, compared to just 0.8% in the baseline group without skills (and only 0.4% in the raw execution log group). The most common scenario is blindly copying steps that seem plausible but don’t fit the current context. This added drag from introducing ill-fitting guides is a textbook “misuse tax”.
The key to maintaining a skill library lies in keeping semantic boundaries clear, rather than blindly chasing complex retrieval algorithms: promptly retire obsolete files, merge overlapping rules, and ensure each skill maintains a single responsibility. Furthermore, explicitly stating in each skill’s description when not to use the guide is the cheapest and most effective way to hedge against this 10.0% misuse tax.
Combining the experimental evidence from the paper with hands-on engineering experience, we can draw three clear boundaries for skills:
First, skills effectively reduce execution jitter, but they cannot replace basic logical reasoning. For high-frequency, tedious engineering tasks like environment setup, format alignment, and background service management, checklist-style step guidance is genuinely effective; but for upfront task decomposition, core algorithm design, and defining what “correct” means, skills cannot do the work for you. Business judgment and acceptance criteria must still be established at the human-led contract layer—skills are merely action stabilizers serving the downstream execution phase.
Second, we must remain prudent about the scope of the experimental conclusions. The evaluation scenarios in this study focus primarily on concrete terminal command and tool-use tasks, without covering long-horizon complex web interactions or open-ended multi-agent collaboration; the model pairings in the experiments only cover two setups, Codex and Gemini CLI—the main Codex experiments were paired with GPT-5.3-Codex, retrieval experiments switched to GPT-5.4 due to model deprecation, and Gemini CLI was paired with Gemini-3.1-Pro-Preview; human categorization and analysis only covered roughly 3% of the compiled execution traces; and this is a preprint published on arXiv that has not yet undergone peer review. When citing these conclusions and numbers, these objective caveats must be kept in mind.
Third, the core value of engineering systems is concentrating further at the contract layer. As platforms standardize skill file specifications and open-source ecosystems mature, the barrier to writing general operating procedures is dropping rapidly, and pure execution stabilizers will steadily become standardized commodities. What truly sets system performance apart will always be the ability to clearly define business goals and design rigorous outcome acceptance criteria—which brings us right back to our core conclusion on result certainty.
Based on these findings, three concrete actions can be put into practice in engineering: 1. Audit your existing skill library, checking paragraph by paragraph whether what you wrote is concrete execution steps or factual statements, stripping out redundant background facts; 2. Distill new skills only from execution logs that carry explicit and trustworthy success/failure labels, avoiding the crystallization of unverified workflows; 3. Regularly prune overlapping and obsolete entries from the library, and explicitly state in the descriptions when that skill should not be used.