When selecting models, many teams claim their solutions are more cost-effective. Yet when deploying coding agents in real-world engineering, the bill at the end of the month remains stubbornly high. Many developers assume that switching to a cheaper model or shortening thinking time will save money, only to see task resolution rates plummet. This discrepancy is common in practice because what people typically call “cheap” actually conflates two entirely different actions.
We can plot a basic coordinate diagram to clarify our thinking. The horizontal axis represents the cost per task resolved, and the vertical axis represents the task success rate. Plotting the measured performance of different models across various configurations, the points that achieve the highest success rate in each cost bracket connect to form a smooth capability boundary. Selecting a point toward the cheaper end of this curve lowers API costs, but at the expense of compromised capability on complex tasks—this is merely point selection via compromise. The other action is pushing the entire boundary outward—tackling harder problems at the same cost, or substantially reducing expenses without sacrificing any success rate. In systems engineering, this capability boundary formed by optimal trade-offs is known as the Pareto frontier.
Mainstream benchmarks have broadly adopted measurements of this performance, such as the Artificial Analysis Coding Agent Index and the DeepSWE Benchmark. On September 10, 2026, Cognition, the team behind the programming assistant Devin, officially released its new model, SWE-2. Vendor-reported data shows that on the same Devin CLI and the same FrontierCode 1.1 Main benchmark, SWE-2’s medium tier achieved higher scores than its predecessor SWE-1.7, while reducing interaction rounds by 58% and cutting costs by 81%. The core technique in its training design is embedding runtime cost directly into the reinforcement learning objective, aiming to make avoiding detours an intrinsic, self-directed trade-off for the model. If pushing the boundary outward is what truly constitutes being cheap, which layer of the system ultimately determines this boundary?
To answer what determines the boundary, we must first look at where the cost of running a programming task is actually decided. In day-to-day engineering practice, teams typically attempt to control budgets across four peripheral layers: input prompting, inference generation, framework orchestration, and commercial pricing. All of these approaches operate outside model weights, and each comes with distinct engineering trade-offs.
The first thing engineers usually try is input prompting. We specify word limits in system instructions and repeatedly remind the model to keep answers concise and rein in divergent thinking, hoping prompts can rein in exploration steps. The TALE prompt budgeting study reported that this practice reduces output tokens by an average of 67%, and options like the Qwen3 thinking parameter configuration also allow disabling thinking mode entirely. Yet textual reminders are ultimately only soft constraints. Faced with thorny code errors, the model still digresses at length, and unnecessary tool calls remain difficult to curb.
Since prompts cannot rein in divergence, engineering intuition turns to hard interventions during generation. Teams monitor thinking token consumption, forcibly terminating thought and shifting to an answer once a preset budget is exceeded—such as the forced thinking budget truncation adopted in the s1 experiment—or adopting Anthropic’s prompt caching feature to lower the unit price of inputs by reusing historical prefixes. While cutting off generation midstream can stop runaway bills, a model whose train of thought has been interrupted often fails to produce a complete patch; the saved tokens come at the cost of task failure.
Because gatekeeping during generation can easily derail tasks, control logic shifts toward external orchestration frameworks. Anthropic outlined context editing and compression mechanisms to clean up historical redundancy from tool calls, and its guide on effective context engineering for AI agents also emphasizes streamlining context; meanwhile, the routing framework RouteLLM offloads simple requests to low-cost smaller models. These routing approaches are essentially shifting tasks across different models, and trimming context risks dropping critical code state without actually reducing the exploration difficulty of any individual inference.
As for flat-rate monthly subscriptions at the commercial tier, they smooth out bill volatility but do nothing to reduce the actual compute consumed by servers. All these peripheral methods merely patch fences outside the model, unable to alter the model’s own reasoning decisions. To push the Pareto frontier outward, the model itself must learn to make trade-offs, injecting cost considerations directly into the training process.
Since peripheral methods cannot push the boundary, the only option is to go inside the model and alter its training process. To enable the model to weigh costs on its own, one must first understand how it adjusts behavior during training. In post-training reinforcement learning, algorithms have the model repeatedly attempt real programming tasks and assign a quantitative score to each rollout. This score evaluating performance is called the reward. The system then converts the score into a directional signal that guides parameter updates—mathematically, the gradient. If scoring only looks at whether the task passed or failed, the policy the model learns may be blindly brute-forcing trials at any cost and repeatedly reading irrelevant files, since getting it right just once by chance yields full marks.
To reverse this habit of disregarding cost, Cognition restructured
the scoring formula during the development of SWE-2, designing the
objective function as R = S − λ_e · C. In this formula, S
represents the rollout outcome—1 for passing the test and 0 for failure;
C denotes the combined expense, converting inference dollar costs and
wall-clock runtime into a single metric; and λ_e is the penalty
coefficient corresponding to different thinking effort tiers. If the
model spends excessive time and tokens on irrelevant exploration, its
overall score shrinks substantially due to the penalty term.
Turning expenditure directly into a penalty gradient stands in contrast to prior approaches. SWE-2 is built on the open-source model Kimi K3. Kimi K3 is an open-source 2.8-trillion-parameter mixture-of-experts model from Moonshot, activating roughly 104 billion parameters per token with a 1M context window. To control costs, Kimi K3 employed hard quotas: a preset token budget cap, where exceeding the budget directly counted as a failure. Rather than using a one-size-fits-all hard cutoff, SWE-2’s continuous penalty is designed to make the model sensitive to the weight of expenditures on every call, spontaneously finding shorter code modification paths.
Vendor-reported data shows that across the same task distribution, the prior-generation SWE-1.7 averaged 127 steps, while SWE-2 compressed this to 53 steps in the medium tier, 80 steps in the high tier, and 98 steps in the maximum tier; the median step count before the first code edit moved from 48 steps in the previous generation to 18 steps in SWE-2’s medium tier. The team attributes this to focused exploration, where the model is far better at homing in on critical code. Yet writing cost into the objective is only the beginning; if the penalty design is even slightly off, the model will quickly learn to exploit loopholes and take shortcuts.
Once cost penalties are introduced into training signals, the greatest challenge is preventing the model from taking opportunistic shortcuts. Reinforcement learning algorithms are exceptionally good at discovering loopholes; if designers fail to think through the mathematical properties of the penalty term, the model will game the system by cutting corners to harvest spurious reward. In practice, Cognition made three key design trade-offs to neutralize three potential slacking loopholes.
The first trade-off concerns the form of the penalty. If a nonlinear
penalty that grows steeper over time is used, a model on long-horizon
tasks will tend to give up and submit prematurely once late-stage
deductions become severe. To resolve this, Cognition derived from
Jensen’s functional equation in Appendix B of its technical
report that if the expected reward across any rollout distribution
is to depend solely on mean cost and mean success rate, the scoring
function must be affine with respect to cost. Normalized, this takes the
form of a simple linear penalty, R = S − λC. The
open-source benchmark OckBench,
by contrast, advocates for a logarithmic penalty, indicating that the
industry has yet to reach consensus on penalty formulations.
The second trade-off concerns setting the penalty coefficient. The same model can operate in multiple thinking modes ranging from frugal to extensive—that is, different thinking effort tiers. If the coefficient is tuned by engineer intuition, the model easily cuts corners across tiers. Cognition’s official technical report revealed failure modes: if the penalty coefficient for a high tier is too large, the model behaves as conservatively in its high-effort tier as in its medium tier; the saved expenses offset the drop in success rate, making the nominal reward appear to rise even though the actual capability boundary never shifts outward. To eliminate this loophole, Cognition anchored the penalty coefficient λ_e to the local tangent slope of the base model’s cost-success curve. This ensures that the iso-reward line is tangent to the Pareto frontier at that point: simply sliding slightly along the curve toward a lower-cost tier has zero first-order effect on reward, encouraging the model to genuinely push the frontier outward to achieve higher gains.
The third trade-off concerns the training architecture for multi-tier models. Conventional engineering often trains multiple expert models separately for different difficulty levels and merges them into a single system via distillation. For example, Kimi K3 trained nine experts and distilled them together, an approach prone to knowledge loss during compression. SWE-2 employs a single reinforcement learning run covering all tiers, teaching a single model to adapt to varied thinking budgets while retaining a length-weighted reward baseline to stabilize gradient variance. Even with thorough safeguards in mathematical formulation and training architecture, this strategy of baking cost into network weights still has clear objective limitations.
Even with rigorous designs to prevent gaming, we must look beyond vendor-reported numbers to examine where this mechanism breaks down and where gaps remain in the evidentiary chain. Synthesizing public benchmarks and third-party experiments, this approach reveals notable engineering limits when facing extreme challenges and long-horizon tasks.
The primary limitation emerges on exceptionally difficult, complex tasks. The e1/AEC paper points out that under a fixed penalty coefficient, when facing problems where the cost penalty exceeds the expected payoff of success, the model’s mathematically optimal strategy is actually to surrender early. Because once it embarks on extensive trial-and-error and search, the continuously accumulating penalties will cannibalize any gains from an unlikely success; to preserve expected reward, the model’s most rational choice becomes giving up without trying.
Another question lies in the attribution of efficiency gains. The official technical report lacks an ablation experiment that removes the cost penalty term; the only ablation comparison is on the computational form of the reward baseline. Compared to its predecessor, SWE-2 adopted the significantly stronger Kimi K3 as its base model. SWE-2’s ability to locate critical code earlier represents an efficiency leap that is difficult to cleanly disentangle from improvements in the base model’s inherent code comprehension. The official report attributes the reduced step count to focused exploration, but fails to substantiate how much of this change depends on the base model’s semantic grasp of code.
On long-horizon tasks, the limitations become even more pronounced. On Terminal-Bench 4, a benchmark involving extensive system interactions, Cognition’s self-reported numbers show SWE-2 achieving only a 27.3% pass rate, trailing Claude Fable 5.1 at 55.8% and GPT-6 Astra at 57.9%. This disparity reflects cross-model and cross-harness performance differences and cannot be taken as a causal conclusion. It suggests that for deep-water tasks with intricate steps, simply reinforcing short-path exploration may not be enough to bridge gaps in complex reasoning ability.
On the proprietary FrontierCode 1.1 Main benchmark, SWE-2 scored a 50.0% pass rate, with the vendor advertising that each rollout was on average 64% cheaper than Claude Fable 5.1. However, this benchmark does not disclose task details, nor does its evaluation harness factor cost into final scores, making it difficult for third parties to independently reproduce the results.
Furthermore, discussing cost figures in isolation from testing environments easily unmoors metrics from any meaningful reference frame. Independent research analyzing test environment effects found that switching external scaffolding on the same set of tasks could cause token consumption per solved problem to vary by roughly 40x. Only by recognizing these objective boundaries can we soberly evaluate the true substance of this achievement.
Having clarified the concrete mechanisms and practical limitations of controlling expenditure at the training layer, we can objectively evaluate the substantial shift this exploration brings to agent development. Moving from peripheral patches to core evolutionary design marks a crucial step forward for software engineering in managing resource consumption.
Looking back at past engineering practices, the burden of controlling costs fell squarely on developers. When calling models day to day, engineers often acted as budget administrators: repeatedly fiddling with thinking effort knobs in request parameters, hardcoding strict word limits into system prompts, or maintaining cumbersome routing and context pruning logic in gateway middleware. In that paradigm, models lacked any awareness of compute resource costs, leaving cost reduction entirely dependent on external layers of defense.
The path illustrated by SWE-2 attempts to imprint resource conservation directly into the model’s network parameters. By establishing continuous cost feedback during reinforcement learning, its training objective encourages the model to turn restraining inefficient trials and prioritizing key code into unprompted habits. By making expenditure an internal trade-off, this approach aims to reduce excessive systemic reliance on peripheral engineering maneuvers.
This shift in training paradigm also poses a deeper engineering dilemma for the entire field. The core appeal of agents lies in their autonomous exploration of unfamiliar challenges; if penalties on compute costs are overly rigid, they risk prematurely extinguishing the sparks of deep reasoning. How to prevent aimless resource squandering while preserving ample trial-and-error runway for complex logical breakthroughs remains a question the industry must continue to answer on the road to practical adoption.