A recent paper benchmarking coding agents ran a particularly fascinating experiment. The authors took the control software surrounding LLMs (commonly referred to as the harness, the layer that passes tools to the model and accumulates dialogue history) and decoupled it into several independent toggles to test just how much each design choice actually matters. When the research team stripped away all built-in file read/write tools and swapped in a pure command-line environment, one LLM’s success rate on code repair tasks jumped by nearly four percentage points, while its estimated API cost was cut by more than half. Yet when tested on another frontier LLM, the success rate plummeted by 23.2 percentage points. The study doesn’t jump to simplistic conclusions. In the authors’ view, harness design comes down to concrete engineering trade-offs: pick the right configuration and you can slash estimated costs in half; misjudge the preconditions and copy-paste the same stripped-down setup, and your execution will completely fall apart.
When building coding agents, our intuition often leans toward subtraction. The common mental model goes something like this: the thinner and leaner the outer wrapper, and the fewer tools placed before the model, the less likely things are to go sideways during interaction. Things like task planning or context summarization, even if not particularly useful day-to-day, at worst waste a bit of compute without actively derailing the problem-solving trajectory. When optimizing systems, many teams naturally treat cutting tools as the default playbook, assuming that lightening the model’s cognitive load can never go wrong.
Real-world benchmarks, however, completely shatter this intuition. No single component in the outer software layer should be treated as an infrastructure default that is turned on without question. Anyone can strip things away; the real challenge lies in figuring out under what exact conditions to do so. If you trim blindly without considering the underlying model’s habits and the nature of the task at hand, the few API calls you hoped to save will quickly turn into an execution disaster.
When we prompt an LLM to write code in a terminal, every action actually runs inside this outer host environment. The LLM itself only receives context and emits text completions; it can neither directly inspect source files on disk nor trigger real system calls in the OS on its own. It relies entirely on the outer software to expose tool interfaces, pass its output commands to the system for execution, and collect stdout and errors from the terminal. Over multiple interaction turns, the harness also stitches these back-and-forth exchanges into dialogue history and manages the constrained context window. In everyday engineering, teams routinely equip models with a full suite of task-planning prompts, external retrieval mechanisms, and dedicated file-editing tools, assuming that more features make the system more robust. Rarely does anyone pause to isolate and measure whether a specific module is actually earning its keep.
A research team from UMass Amherst, Emory University, and UNC Charlotte released an empirical study in September 2026 (preprint at arXiv:2609.20804, full text at the arXiv HTML page). Holding the standard think-act-observe loop constant, they decoupled the outer software into three independently toggleable core modules, measuring the real-world performance of each module piece by piece.
The first switch controls task planning. When this switch is toggled on, the system prompt explicitly instructs the model to draft an execution plan before tackling complex tasks. With each subsequent step, the harness re-injects the latest plan into the prompt and provides a dedicated tool called update_plan so the model can update its progress at any time. If this switch is toggled off, both the planning instructions in the prompt and the update_plan tool are completely removed, and the model dives straight into the task upon receiving the objective.
The second switch governs context management. As execution stretches across many turns, the researchers tested five strategies for handling history. The crudest approach is to do nothing: let history accumulate until it exceeds the context window and terminates. A slightly more refined approach uses hard rule-based truncation, replacing older tool outputs with stub placeholders once history crosses a hard threshold while preserving structural integrity. The third, more sophisticated approach stores pruned long text in external storage and provides a dedicated tool allowing the model to retrieve historical details on demand. The fourth approach tasks the tested model itself with clean-up: once history crosses the hard threshold, a separate tool-free call distills prior key points into a condensed summary. The final approach is a staged hybrid strategy: rule-based pruning kicks in as soon as the context exceeds a soft threshold; if it still exceeds the hard threshold after pruning, the same model is invoked to generate a summary.
The third switch controls the action execution interface, meaning the actual tools at the model’s disposal. Under the conventional full-tool setup, the harness equips the model with eight handy dedicated tools spanning file reading, file writing, patch replacement, directory traversal, regex search, and terminal command execution. But when switched to pure command-line mode, the harness strips away all predefined file and search tools. The model is left with only raw terminal commands, relying entirely on running shell commands to navigate the workspace and edit source code.
To observe how models across different tiers react to these switches, the researchers evaluated them on two benchmarks. One is the familiar Python code-repair benchmark SWE-Bench Verified, comprising 500 real GitHub issues, designed to test a model’s ability to locate files, analyze logic, and make precise edits across large codebases. The other is Terminal-Bench 2.1, an end-to-end command-line benchmark with 89 terminal-native tasks that directly evaluates operational proficiency in configuring services and troubleshooting system states in a shell environment. Evaluated models spanned three sizes of the NVIDIA Nemotron-3 family (30B, 120B, 550B), alongside the open-weights frontier model Mistral-Medium-3.5-128B. Crossing five context strategies with four window sizes (32k, 64k, 96k, 128k), plus ablation experiments for planning and interfaces run exclusively under staged management with a 128k window, the team ran a total of 176 experimental configurations. Per-task costs were estimated using OpenRouter August 2026 token pricing, and evaluations employed paired success/failure testing with false-discovery rate corrections, running each configuration once per problem.
To properly interpret the findings, two crucial engineering implementation details from the paper are worth noting so we don’t draw the wrong conclusions. First, switching to command-line mode is a bundled substitution. While removing dedicated file tools, the authors simultaneously adjusted the system prompt, removing pre-write read checks, pre- and post-edit state tracking, and automatic diagnostic prompts following supported edits. This not only changed the number of tools available to the model, but also dismantled the harness’s surrounding guardrails. Second, pure command-line mode does not mean literally only a single terminal command remains in the entire environment. When planning is enabled and staged context management is used, command-line mode still retains the plan-update tool and the history-retrieval tool; the harness’s task-tracking mechanisms are not completely dismantled.
Once the harness was taken apart, the starkest contrast emerged when switching the action interface. Under the baseline setting with a 128k window and planning enabled, the researchers removed the dedicated file tools and switched to pure command-line mode. For the 550-billion-parameter Nemotron-3 550B on the SWE-Bench Verified code-repair benchmark, task success rate climbed from 65.80% to 69.40%, a gain of 3.6 percentage points, while average estimated cost per task dropped by more than half, from $2.33 to $1.11 (see the paper’s Table 3 results table). Yet when the exact same streamlining was applied to the fellow frontier model Mistral-Medium-3.5-128B, its success rate plummeted from 68.60% to 45.40%, a drop of 23.2 percentage points. While its estimated per-task cost also fell from $3.14 to $1.72, its problem-solving capability collapsed. Statistical tests showed both differences were statistically significant.
To unpack why the two models diverged so dramatically, the researchers used an LLM judge to inspect the execution trajectories. Human spot-checks covering over 15,000 annotation units across 200 trajectories, including action tags and failure-stage diagnoses, showed an agreement rate of approximately 94.2% between the judge and human annotators, with a weighted mean Cohen’s κ of 0.929, indicating high reliability. Looking closely at the detailed trajectory logs, the genuine difference between the two models in the command-line environment becomes immediately clear.
Nemotron-3 550B’s trajectories demonstrated sophisticated command-orchestration chops. Without standalone file tools, 550B did not flinch, condensing file discovery, regex matching, and code replacement into compact shell scripts and one-liners. Benchmark data shows that on code repair, 550B’s median maximum edit size in command-line mode jumped from 18 lines to 54 lines, while repeated edits to modified files plunged from an average of 4.6 times per task to 1.5 times, reducing tool calls by 32%. By replacing iterative trial-and-error with high-density single executions, it drastically reduced interaction turns and solved problems far more decisively.
Mistral-Medium-3.5-128B’s trajectories told an entirely different story, riddled with localization failures. Once dedicated file tools were pulled, Mistral struggled to locate target files within nested directory structures. In command-line mode, the proportion of tasks where Mistral exited without submitting any code edits reached 32.80% (compared to just 1.20% under the full-tool setup); among unresolved tasks, the rate of failing to even touch the correct file surged from 16.00% to 41.40%. Without the read-before-write checks and edit diagnostics provided by the harness tools, the model drifted off the remediation path early during search and script assembly.
Switching to a different task setting, however, brought an immediate plot twist. On Terminal-Bench 2.1, with its 89 terminal-native tasks, switching to pure command-line mode actually boosted Mistral’s success rate from 37.08% to 43.82%, an increase of 6.74 percentage points; 550B similarly climbed from 44.94% to 50.56% (see the paper’s Table 4 results table). Although neither gain cleared the statistical significance threshold given the small sample size of 89 tasks, both trends pointed in the same direction. The action logs reveal why: in terminal-native tasks, even with the full tool suite available, Mistral proactively routed 71.90% of its workspace actions through the command line, compared to only 40.40% in code repair tasks.
This comparison lays bare the fundamental nature of harness design: outside of specific contexts, there is no universal best practice; all performance shifts with operating conditions. How much value an outer component delivers depends entirely on how well the model’s command-line habits align with the native characteristics of the task. Copying the exact same stripped-down setup unlocks high-efficiency autonomous execution in one model, but tears away the indispensable guardrails keeping another upright.
Examining the three modules tested in the paper one by one clarifies the true boundaries under which each operates. Starting with the first knob, task planning, experiments demonstrate that planning mechanisms are a crucial lifeline for weaker models. Take the 30-billion-parameter Nemotron-3 30B: turning planning on boosted its code repair success rate from 13.60% to 25.20%, a substantial gain of 11.6 percentage points, while estimated cost per task only nudged up from $0.02 to $0.09. But once planning was switched off, 30B’s median interaction turns crashed from 40 to 5; 68.60% of tasks yielded zero edits from start to finish, and 58.40% got stuck entirely during code localization. The externally injected planning structure sustained the execution chain, allowing the weaker model to hold on until it could issue its first edit.
For mature, highly capable models, the role of planning shifts immediately: it stops boosting accuracy and instead serves primarily to manage cost and clamp down on runaway spending. For both capable models, 550B and Mistral, enabling planning did not improve accuracy: 550B’s success rate moved from 67.80% to 65.80% (down 2.0 percentage points), while Mistral edged down from 69.00% to 68.60% (a slight dip of 0.4 percentage points). However, the cost savings from planning were substantial: estimated per-task costs dropped by about 30% for 550B (from $3.31 to $2.33) and by about 32% for Mistral (slashed from $4.65 to $3.14). The trajectory logs explain everything: strong models don’t need external instructions telling them how to find code; the core value of planning lies in reigning in redundant verification. Without planning, 550B would endlessly re-check its work after completing changes, blowing up median interaction turns from 74 to 108, while Mistral’s turns expanded from 53 to 68. In this regime, the planning mechanism morphs from a capability booster into a cost-control valve. Real-world industry engineering reflects the exact same trend: issue #80487 in Anthropic’s official repository reveals that Claude Code disabled task creation and checklist tools by default for newer models like Opus 4.8, Sonnet 5, and Fable 5, keeping only an environment variable for manual activation. This shows that as foundation model capabilities advance, structured planning is steadily receding into an on-demand optimization toggle.
The second knob centers on context management. Experimental data indicates that the primary benefit of managing context is preventing window overflow; its impact on improving step-by-step reasoning quality is actually quite limited. As the physical window expanded from 32k to 128k, the average success rate gap between managing and not managing context narrowed rapidly: in code tasks, the gap shrank from 35.7 percentage points at 32k down to 2.7 percentage points at 128k; in terminal tasks, it fell from 9.5 percentage points to 2.8 percentage points. Under a cramped 32k window, leaving context unmanaged led to an average of 78.70% of code tasks terminating prematurely due to window overflow; enabling management brought the overflow rate to zero across the board. Across all 8 tested model-and-benchmark combinations, the staged strategy achieved the lowest cross-window average estimated cost among all management strategies in 7 of them, proving that prioritizing hard-rule truncation and only invoking the model for summarization when limits are breached combines both stability and cost-effectiveness.
The retrieval mechanism designed specifically around context yielded surprisingly lackluster results. The premise of this design was to offload long text into external storage during pruning, allowing the model to retrieve past details on demand. Yet across all 32 pairwise comparisons, the retrieval mechanism notched only 15 wins, 14 losses, and 3 ties, yielding an average accuracy delta of -0.36 percentage points. Out of 64 configurations with retrieval enabled, a full 36 never called retrieval a single time from start to finish; under the 128k window, it was called an average of just 0.007 times per task. Engineering such a complex external storage-and-retrieval pipeline rarely produces meaningful returns; in actual execution, models rarely take the initiative to backtrack through old outputs previously pruned by rule.
The third knob governs the action interface. Nemotron-3 30B’s performance on the terminal benchmark serves as a classic counterexample: after switching to pure command-line mode, 30B’s success rate collapsed from 13.48% to 3.37% (see the paper’s Table 4 results table). Log analysis revealed that a full 66.00% of its trajectories failed because it attempted out-of-interface tool calls. The model continued emitting the specialized function call syntax memorized during training, failing to realize that the active environment offered only a shell; its average interaction turns plummeted from 71 to 15 as a result. While strong models can indeed reap benefits from streamlined interfaces, defining that prerequisite in advance remains tricky. As commenter agentdev001 pointed out, the entire paper emphasizes that a model must possess command-line proficiency, yet never provides an operational, testable definition beforehand, a critical prerequisite that practitioners usually can only reverse-engineer after tests have already crashed.
However, when interpreting these conclusions, one must also account for the specific experimental setting. The authors candidly noted that they tested only a single planning prompt and one threshold scheduling strategy; all interface ablation experiments were conducted strictly on top of a 128k window with staged management; and code tasks were restricted to Python and evaluated on single runs. Community discussions have been equally divided. Commenter vblanco questioned whether open-source models can accurately represent commercial closed-source frontier models that undergo heavy alignment for specific scaffolds and boast larger native context windows. Commenter Systemerror7A69 pushed back, arguing that in the absence of empirical evidence of equal rigor, one cannot simply wave away the empirical data at hand. This debate serves as a healthy reminder that findings from the paper cannot be blindly extrapolated across different model families.
Faced with this body of empirical data, there is no need to swing to the extreme of dismissing harness architecture altogether. The real takeaway is shedding the default assumption that every outer mechanism should be turned on, and treating harness design for what it genuinely is: an engineering trade-off tailored to specific operating conditions. When building coding agents yourself, three critical decisions demand your attention.
First, assess the foundation model: verify whether its independent ability to script and navigate files is genuinely up to par. If the chosen model lacks sufficient training on terminal interactions, frequently stumbling into syntax errors or getting lost in directory paths when dropped into a command line, you need to conscientiously equip it with dedicated file tools, relying on the tool layer to safeguard a baseline success rate. Conversely, if your model already possesses mature scripting habits, pruning away dedicated tools gives it room to run freely while substantially slashing interaction turns and API overhead.
The second decision hinges on the task environment: weigh carefully what fraction of the workload inherently relies on the shell. If the workload is centered on operations, environment configuration, or other shell-centric domains, going pure command-line for the action interface is by far the cleanest route; wrapping it in dedicated tools often just introduces unnecessary redundancy. But if the task involves refactoring source code in a large, complex repository, the read-before-write checks, crisp edit boundaries, and timely diagnostic feedback provided by dedicated tools serve as sturdy guardrails against blind, erratic codebase edits.
The third decision comes down to the ledger: calculate how deep your context management can afford to go given your window budget and API costs. If your context headroom is generous and budget is plenty, basic overflow prevention is all you need, no fancy footwork required. But under tight budgets or long-horizon executions, a staged strategy, prioritizing rule-based trimming and only invoking the LLM for summarization when thresholds are breached, is the premier choice for cost control. As for complex external retrieval mechanisms, they simply aren’t worth the trouble to maintain.
Viewing tool selection through the lens of key decisions also helps clear up a widespread misconception in the community. When the paper first dropped, commenter jimbokun suggested that if the command line is this powerful, specialized tool protocols like MCP might become obsolete. Commenter CharlieDigital stepped in to clarify that specialized tool protocols and local command lines solve problems at entirely different layers. The core value of structured protocols lies in enterprise credential management, fine-grained permission isolation, and remote environment interaction. Real-world production systems will not expose a raw host bash terminal to every external invocation; the permission boundaries and auditability offered by structured interfaces simply cannot be replaced by a raw command line.
Academic research over recent years similarly shows that harness construction always hinges on concrete engineering decisions. Liu’s 2026 evaluation of prompt components (see arXiv:2605.05716) found that turning on all components rarely yields the optimal solution across most scenarios, and that certain negative effects actually flip to positive as model scale increases. An empirical study by Mehtiyev and Assunção evaluating 19 agents across 8 frameworks (see arXiv:2604.02547) showed that performance gaps between frameworks continue to narrow with each model generation. Research by Yang et al. in July 2026 on code execution interfaces (see arXiv:2607.10569) reported that determining which tool interface is most cost-effective ultimately depends on both task characteristics and system architecture.
Finally, consider an open question stemming from a community hypothesis. Commenter svachalek noted that massive volumes of real interaction trajectories are being baked into the training pipelines of next-generation models, gradually internalizing these outer behavioral patterns into innate instincts. If that proves true, how many items on today’s harness decision checklist will engineers still need to weigh tomorrow?