The official benchmarks for DeepSeek V4 Flash are undeniably impressive: an MoE architecture with 13B activated parameters, priced at a fraction of comparable models, scoring 82.7 on Terminal-Bench 2.1, 70.3 on Toolathlon-Verified, and 76.7 on Cybergym. But when the exact same model is placed into different runtime environments, its performance on the identical set of tasks fluctuates dramatically.
In August 2026, Composio connected DeepSeek V4 Flash to 8 different agent harnesses to test the same 30 SaaS tool-calling tasks. With the model completely unchanged, Pi Agent solved 20 tasks, achieving a 66.7% win rate, while OpenCode only passed 14, at a 46.7% win rate. The exact same model saw a 20-percentage-point difference in pass rate simply from switching the harness. (36kr covered the benchmark concurrently.)
Similar discrepancies have emerged in third-party replications. When Artificial Analysis independently reproduced DeepSeek V4 Pro’s Terminal-Bench 2.1 results using their own harness, they scored only 79—9 points lower than the officially reported 87.9. In analyzing this gap, Cloud Codes pointed out that the 87.9 score ran on DeepSeek’s own harness, whereas the 79 ran in an external environment. Benchmark scores have never been a reflection of model capability alone, but rather the composite output of a model combined with its scaffolding. On Hacker News, someone asked directly: “Are we benchmarking the model or the harness?”
Why does switching scaffolding lead to such a huge performance gap? The answer is actually hidden in DeepSeek’s own API changelog. Below the official benchmark table is a single footnote: all coding agent benchmarks were run on DeepSeek Harness’s minimal mode. This stripped-down mode provides only two persistent tools—bash and str_replace_editor—without web search, skill libraries, sub-agents, or planning modules. In this carefully pruned minimalist environment, the model performs smoothly; but once it leaves this setup, test pass rates fall. While agent performance depending on the harness is common industry knowledge, this footnote exposes a much more specific issue: the model’s high-scoring performance is tightly bound to a specific minimalist runtime environment.
How DeepSeek reached this point involves both technical path choices and product format constraints. First is the inertia of its technical DNA. Starting with R1-Zero, DeepSeek shifted fully toward synthetic data, using data generated in math and coding to enhance model reasoning, proving that strong reasoning capabilities could be trained without large-scale human annotation. These single-turn tasks have clear objective criteria: math problems have unique solutions, and code execution correctness can be verified directly. The success of R1-Zero established this synthetic data methodology. By the V4 Flash 0731 release, post-training continued along this path: partitioning expert models by domain, training each expert with SFT and GRPO, and finally merging and distilling them. GRPO is a group-relative reinforcement learning algorithm that compares relative advantages based on rewards directly across multiple sampled completions within the same group. This algorithmic trait is directly tied to the environment dependencies exposed later.
This synthetic data playbook works for single-turn tasks because math problems have standard answers, code execution has objective verification, and synthetic data can include its own verifier. But multi-turn agent tasks are different. A customer support agent needs to converse back and forth with users across multiple turns; a coding agent must repeatedly execute tools, inspect error messages, and make revisions. Its reward comes from the entire sequence of interaction, with no standard answer determinable in advance. In this setting, the user behavior or tool feedback in the training environment directly determines what policy the model learns—the behavioral distribution of the environment is the training distribution. Synthetic data can generate tasks, but it is very difficult to generate interaction trajectories diverse and realistic enough to cover all kinds of user responses.
This leads to DeepSeek’s data dilemma: it lacks large-scale, authentic user behavioral trajectories. OpenAI has Codex, Anthropic has Claude Code, and Cursor has its own IDE—these products serve as execution environments for agent tasks and entry points for collecting real-world coding interaction data. How users clarify requirements, iterate on code based on compiler errors, when they decide to give up, and which tasks require multi-turn iteration in practice—all these behavioral trajectories accumulate in the vendor’s infrastructure, serving as real-world foundation for refining model agent capabilities under compliant conditions. Subscription products like ChatGPT and Claude cannot collect authentic interaction data from coding scenarios on their own; a chat interface and an agent execution environment are two entirely different things. DeepSeek had neither, with the latter remaining a long-standing blank. Its services primarily run through APIs, lacking proprietary end-user products and interactive interfaces for a long time. As we previously analyzed regarding the shifting form factors of coding agents, DeepSeek held off on releasing a first-party harness, conceding the behavioral data entry point in execution environments to third parties. It was not until August 13, 2026, with the release of DSH (see timeline on orcarouter), that DeepSeek finally had its own harness. When developers build agents with its models, all behavioral trajectories stay inside third-party tools like Codex, Claude Code, or OpenCode, and the data never flows back to DeepSeek.
A clarification is warranted here: that lacking proprietary end-user products prevents direct collection of large-scale real user behavioral data is a logical inference based on DeepSeek’s product form factor, not an official statement from DeepSeek. But the product form factor is a public fact, and the inferential chain is short.
Technical inertia and data constraints thus formed a loop: reliance on synthetic data slowed the pace of building a proprietary harness to collect real data, while the scarcity of authentic behavioral data in turn forced training to continue relying on synthetic environments. Once multi-turn interaction became the main battleground of agent training, this bottleneck was exposed.
On August 12, 2026, a joint team from Stanford, UC Berkeley, Northeastern, NYU, and UW published the paper “One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL”, revealing a collapse mechanism called simulator collapse that neatly explains DeepSeek’s harness dependency. The paper points out the core flaw of single-simulator training: when using only a single LLM as a user simulator to train an agent, the policy evolved by the model finds shortcuts specifically targeted at that simulator’s particular response patterns, ultimately losing the ability to adapt to diverse human behavior.
Aligned LLMs widely exhibit a tendency toward mode collapse, easily suppressing tail distributions to output high-frequency stereotypical replies. When using such a model as an environment simulator, the feedback it provides at every interaction step is highly homogeneous. The agent’s training gradients therefore concentrate on this dominant mode, and optimization gradually shifts toward scoring high under fixed responses. As reinforcement learning training continues, the diversity of interaction trajectories drops rapidly. Tracking the training process, the research team found two clear signals: within the same batch of samples, rewards across entire dialogue groups grew increasingly uniform, eventually leading to over 85% of batches where all samples received identical scores, providing zero discriminative signal for the model to learn from; at the same time, policy entropy dropped from 1.9 nats all the way down to 0.4 nats, meaning the policy produced virtually only a single dialogue trajectory.
This collapse causes a divergence between training metrics and generalization capability: even though the model’s reward in the training environment keeps rising, actual win rates drop whenever it is switched to an unseen simulator or a real human scenario. The research team conducted a human experiment involving 320 Prolific sessions, tasking the agent with persuading users to donate. The results showed that agents trained on a single simulator raised an average of only $0.46 in donations, which was actually lower than the unreinforced baseline model ($0.51).
The paper evaluated simulators across three different families—GPT-5-mini, Haiku-4.5, and Gemini-3-Flash—and observed the collapse across all of them. Even when expanding the policy model size from 4B to 8B, the collapse still occurred. This demonstrates that the bottleneck lies not in the absolute size of the model or simulator, but in the width of the environment’s behavioral distribution.
The collapse mechanism discovered in the paper corresponds directly to DeepSeek’s benchmark performance. The DeepSeek V4 Flash 0731 release used the GRPO algorithm in post-training. At the core of GRPO is group-relative advantage, which calculates relative rankings across a group of sampled sequences by normalizing reward mean and variance. This shares the same algorithmic logic as the within-group reward normalization mechanism used in the simulator collapse paper.
Based on public evidence, a reasonable explanation can be derived: the 0731 update only changed the post-training stage without altering the base architecture and parameters, and the official benchmark tests were all run on DSH minimal mode, which only provides bash and str_replace_editor. If the agent loop’s actual operating mode during post-training heavily overlapped with minimal mode, lacking web search, sub-agent collaboration, and complex planning, the optimal policy learned by the model would be deeply bound to this specific environment. When running in the official harness, tasks remained within the training distribution; once switched to third-party harnesses and the environment changed, the model fell out of distribution.
To be explicitly clear, the logic above regarding harness overfitting is an inference based on public evidence, not a confirmation from DeepSeek. Without public training logs, the exact configuration of the agent loop during post-training cannot be definitively determined. However, given that 0731 only modified post-training, official top scores were tied to minimal mode, third-party reproduction scores declined, and the paper validated the collapse mechanism, this deduction is supported by a coherent chain of evidence.
When a model’s scores drop after changing environments, the intuitive fix is often to upgrade the test container by enabling Standard mode with search, sub-agent, and planning capabilities to pull scores back up. The simulator collapse paper rejects this simplistic replacement approach. Research shows that from GPT-5-mini to Gemini-3-Flash, simulators across different families triggered collapse without exception. The root of the problem is not whether the simulator itself is powerful, but that its behavioral response distribution is too narrow. Aligned LLMs come with an inherent tendency toward mode collapse; simply switching models cannot resolve the issue of homogenized output patterns.
The same applies to harnesses. If the model was only exposed to minimal mode behavioral trajectories during training, switching harnesses during the inference phase merely changes the degree to which the overfitting problem is exposed, without resolving the distributional bias left behind during training. Even if scores measured under Standard mode might be higher than OpenCode, it is only because Standard mode’s tool configuration is slightly closer to the training conditions. This remains a localized extension of the original training distribution, not an independent test of generalization.
To solve this problem, the paper proposes two remediation paths, both centered on broadening the environment’s behavioral distribution rather than adjusting policy-level exploration scheduling. The first method is introducing Verbalized Sampling at inference time. It requires the simulator to output a list of candidate texts and their corresponding probability distribution before generating a response, and then sample from it. Doing so forcibly brings out atypical responses that originally had lower probabilities and would easily be ignored by the model, restoring diversity to environment feedback. This mechanism does not require retraining the model, adds only decoding computation at inference time, and improved task success rates in held-out scenarios by up to 9%.
The second method is introducing Co-Training during the training phase. By updating the simulator and policy model in sync, the simulator’s response characteristics drift continuously throughout training. Shortcuts previously found by the policy model become invalid as the environment changes, forcing the policy to maintain generalization ability. The paper also designed an enhanced version called Population Co-Training, which randomly samples from the 5 most recent simulator checkpoints as the opponent environment in each training round, ultimately improving held-out success rates by up to 14%.
What these two approaches share is that both broaden the feedback distribution of the training environment. This also accurately reflects DeepSeek’s previous shortcomings. Before having a first-party harness, the feedback distribution of DeepSeek’s training environment was shaped entirely by synthetic data, with its upper bound limited by the coverage of synthetic rules. Releasing DSH opened a channel for collecting real user behavioral trajectories and broadening the distribution; however, because this engineering effort just went live, data accumulation has only just begun.
Revisiting DSH through the framework of the simulator collapse paper provides a deeper understanding of its architectural design. Previously, we analyzed DSH in detail, highlighting its hot-swappable agent loop feature as a form of self-evolving infrastructure. But viewed from the perspective of multi-agent reinforcement learning, this design is essentially the engineering realization of Co-Training principles at the harness architecture level.
The core logic of Co-Training is to co-evolve the training environment and policy model together, preventing the policy from overfitting to a single environment response pattern. DSH decouples the agent loop into plugins that can be dynamically loaded at runtime, allowing the training environment to adjust alongside policy iterations. If DeepSeek incorporates DSH’s diverse agent loop configurations into future post-training pipelines rather than remaining confined to minimal mode, it is effectively practicing Co-Training at the environment level.
The value of open-sourcing DSH (code on GitHub) goes beyond making it easier for the community to replicate existing benchmarks. Replaying tests in an already overfitted environment will still yield biased scores. Its true significance lies in making public the environmental structure on which model training relies, opening it up for collective auditing and expansion by the community. By adjusting agent loops within DSH and observing score drifts, developers can decouple intrinsic model capabilities from the auxiliary role of scaffolding—an essential step for diagnosing and correcting policy collapse.
These experiments and theoretical frameworks provide agent R&D teams with a clear diagnostic rubric. When an agent’s reward steadily rises in a specific training container, but scores drop significantly as soon as it switches harnesses, connects to different user groups, or shifts task distributions, it indicates that the system has fallen into simulator collapse. In this case, troubleshooting should not blindly focus on tuning policy exploration algorithms or expanding model parameters, but rather on broadening the behavioral distribution of the training environment.
For DeepSeek, their mindset underwent a transformation: from initially believing synthetic data could cover the entire lifecycle, to recognizing that the distribution width of the training environment is the ultimate barrier. This shift prompted them to move beyond pure model R&D toward building harness infrastructure. The original motivation behind developing DSH was the realization that the core bottleneck constraining agent performance improvements has shifted from model architecture to data distribution. To break through this limitation, taking control of the harness into their own hands became essential. Competing with Codex for the end-user market was merely a secondary byproduct.