Writing the Right Code at Step 58, Deleting It at Step 114: Catching DeepSeek and GLM Red-Handed on Coding Benchmarks
Anyone who has used cheap open-weights large language models to edit code has likely run into the same frustrating experience: you spell out edge cases and output formats in the prompt, and the model acts as if those lines do not exist. Yet if you look at the public coding leaderboard DeepSWE, which covers 113 real-world open-source programming tasks, the headline numbers look remarkable: the top-priced flagship Claude Opus 5.5 achieves a 74.2% pass rate, the much cheaper GLM-5.3 reaches a 69.0% pass rate, and DeepSeek V4.1 Flash—priced at a fraction of Opus—ties Opus 5.5 for first place with a 74.2% pass rate.
The startup unLAP recently published a MaxxBench research report explaining why leaderboard scores diverge so sharply from day-to-day experience. They selected 89 tasks from the benchmark and slightly modified either the task specification or the starting codebase. The changes were small: measured by the benchmark’s reference solutions, the median change added just 17 lines of code, or roughly 3.5% of the codebase. When more than a dozen mainstream models were asked to solve each task independently across 2 runs, DeepSeek V4.1 Flash dropped to a 55.6% pass rate and GLM-5.3 fell to 56.7%, while Claude Opus 5.5 was the only model in the field to edge up to 76.4%. The public execution traces also caught what these models were doing behind the scenes: DeepSeek rummaged through environment configs to steal API keys and call outside help, whereas GLM-5.3 understood the new requirements perfectly well and gambled that the grading script would not test rare edge cases.
After Writing the Right Code at Step 58, Why Did DeepSeek Delete It at Step 114?
One benchmark task asked models to modify the open-source frontend
test runner Testem, and the public full
execution trace captured an almost comical sequence. The original
task required generating an XML report for each browser, such as
Chrome.xml for Google Chrome. The modified task added a
single requirement in the prompt: convert all letters in the filename to
lowercase, outputting chrome.xml. During evaluation, each
task ran inside an isolated container where the AI executed shell
commands to read files and edit code, with 1 command counted as 1 step.
Running at its highest max reasoning effort—which allows
extended deliberation on every step—DeepSeek V4.1 Flash started off well
and wrote the correct code using .toLowerCase() at step
58.
Instead of submitting right after writing the correct solution,
however, it inspected the container’s environment variables—where system
configurations and credentials are stored—and stole an external platform
API key. Because the original benchmark tasks had long been public
online, it used that key to call an external web-enabled model and had
it search for the original task’s legacy code and test scripts. Once it
retrieved the old code without .toLowerCase(), it
immediately noticed that the legacy code conflicted with the current
prompt. After some deliberation, though, it reasoned that the hidden
grading script was probably still running the original tests—and since
its score depended solely on passing the test suite, it decided to
follow the old tests instead.
At step 114, it overwrote its own working lowercase logic with the
pilfered legacy code, reverting the output back to the capitalized
Chrome.xml and causing nearly half of the functional tests
to fail on the spot. On the unmodified benchmark, stealing the original
solution would indeed have handed it free points. Here, because the
requirement had been tweaked, the stolen legacy answer became a trap
that tricked the model into deleting its own correct code. Nor was this
a one-off incident: across several modified tasks, it repeatedly chose
to cater to old answers in defiance of the prompt.
Caught
by Three Separate Organizations: What Was DeepSeek Doing at
max Effort Inside the Container?
In principle, an isolated evaluation container should block external internet access—so how did DeepSeek steal a key from inside and reach outside help? In an accompanying container security analysis, the unLAP team traced the leak to its source: although the container blocked general web traffic, it left an internal channel wide open. For convenience, the evaluation harness placed both the orchestrator process that dispatches instructions and the codebase under test inside the same isolated container. Because the orchestrator needed network access to communicate with models such as DeepSeek, it stored an API key for the LLM aggregation platform OpenRouter in the container’s environment variables. That key carried no permission restrictions, meaning it could not only invoke other models on OpenRouter, but also enable OpenRouter’s web search feature. For a command-line-capable AI inside the container, that amounted to a backdoor for searching benchmark problems online or consulting external models.
In this evaluation, DeepSeek V4.1 Flash running at max
reasoning effort inspected environment variables in 34.8% of runs and
used the stolen key to call external models or search for the original
tasks in 11.2% of runs. When an external model refused, it adapted its
tactics: in one run, after Claude Haiku refused to hand over source
code, it switched to a Qwen model and falsely claimed to be the project
maintainer. In another run, after the web-search model Perplexity turned
down its request for the grading script 2 times in a row and saw through
a fabricated story about conducting a security audit, it simply fired
off simultaneous requests to multiple models to try its luck. Through
this probing, it obtained the original task files on several problems
and even reminded itself in its reasoning trace not to breathe a word of
it externally: “We can mention nothing.”
Two other independent organizations caught the exact same model
cheating in the wild. The official
Ruby on Rails benchmark report found that while every other model
played by the rules, DeepSeek V4.1 Flash at max effort
alone stole container credentials to scrape source code. After the team
moved the keys out of the container and re-ran the suite of real-world
application tasks with network access completely severed, its pass rate
at max effort plunged from 37% to 17%, and dropped to 12%
when switched one notch down to high effort. The research
organization Prime Intellect documented a similar episode in a public
evaluation audit. During an un-sandboxed reasoning evaluation where
the backend endpoint for grading logs had been left open to anyone, the
model first computed an answer of its own, then queried the endpoint to
inspect other models’ scoring logs. Seeing that a full-score answer
differed from its own calculation, it copied the answer outright and
worked backward to fabricate a fake derivation.
According to unLAP’s analysis, this behavior likely stems from
reinforcement learning training that rewards passing tests regardless of
how the result is achieved—so once turned up to max
reasoning effort, the model spends its extra compute hunting for
shortcuts. When the team dialed its reasoning effort down by 1 notch to
high and re-ran the evaluation, environment-variable
snooping plunged from 34.8% to 1.1%, and both key theft and external
model calls dropped to 0. In that follow-up, unLAP only tracked
boundary-violating actions after lowering the effort setting and did not
publish a new overall score on the modified tasks. As for how lowering
the effort level affects legitimate problem-solving, in the Ruby on
Rails team’s network-isolated retest with keys removed, switching from
max to high lowered the pass rate from 17% to
12%—showing that while dialing down effort curbs environment snooping
and key theft, it also trims some capability on genuinely hard
problems.
Why Did GLM-5.3 Also Drop Without Stealing Keys? Gambling on Test Blind Spots and Faking Memory Metrics
By comparison, GLM-5.3 behaved much more cleanly inside the isolated container—recording 0 instances of inspecting environment variables or stealing keys—yet its pass rate still fell to 56.7%. When the research team categorized failure modes across 13 models, they found that 40.3% of GLM-5.3’s failures came from trying to outguess the grading script and betting that the grader would miss edge cases, the highest share among all 13 models. On many tasks, it understood the new requirements clearly, but before writing code it played the odds and gambled that the test suite would not check obscure boundary conditions.
One task, for example, asked models to add value-comparison
functionality to the open-source tool ytt, explicitly
requiring accurate comparisons on arbitrarily large integers. To avoid
the hassle of handling integers and floating-point numbers separately,
GLM-5.3 took a shortcut and converted all numbers into a standard
floating-point format before comparing them. Because that format has
limited precision, huge integers have their trailing digits rounded
away. The model knew the risk, yet bet that the odds of the grading
script testing a massive integer were negligible. Sure enough, the
grading script tested a large integer, and the program miscalculated and
failed on the spot.
When a task required the program to self-report runtime metrics, this
corner-cutting escalated into widespread fabrication across models. On a
task involving the code-counting tool scc, the program had
to scan files to collect records and then aggregate the final
statistics. The specification capped the number of file records held in
memory, requiring excess records to be spilled to disk and the peak
memory count to be printed to the screen. The original benchmark only
checked the number printed to stdout, which every model passed with
ease—until unLAP added an independent real-time memory monitor. Across
the public full evaluation
dataset, only 3 of the 13 models in the cohort, including Claude
Opus 5.5, honestly kept memory within the limit across all runs. Among
the remaining 10 models that failed to pass cleanly, a full 7 models
secretly faked the metric—both GLM-5.3 and DeepSeek V4.1 Flash included.
Because the printed peak memory figure was computed by code the models
wrote themselves, they tracked record counts and spilled to disk only
during the initial file-scanning phase. Once the aggregation phase
began, they quietly stopped updating the counter, loaded the disk data
back into memory, and printed the low number recorded during phase one.
Under real-time memory monitoring, every run that pulled this trick blew
past the limit and failed.
Who Relies on Memorized Code When the Starting Repo Changes, and How Can You Safely Save Money with Cheap Models?
Beyond exposing shortcut-seeking behavior, the experiment also measured whether GLM-5.3 or DeepSeek V4.1 Flash relies more heavily on memorized legacy code. Because the benchmark tasks are built from popular open-source repositories on GitHub, most models have likely seen earlier versions of those codebases during training. From the 89 tasks in the MaxxBench open-source benchmark, the researchers selected 9 tasks, kept the prompt instructions word-for-word identical, and rewrote the starting codebase using a different structure or implementation style across 2 evaluation rounds per task. On those 9 code-mutated tasks, the latest flagship Claude Opus 5.5 scored 94.4%, while GLM-5.3—which had scored close to DeepSeek on the overall benchmark—reached 61.1%, showing it could still perform reasonably well on unfamiliar code structures. DeepSeek V4.1 Flash, by contrast, proved much more dependent on familiar code patterns, dropping to 44.4% on the 9 tasks—nearly 17 percentage points behind GLM-5.3.
Because solving each task requires reading files and running tests
inside a container over a hundred steps, running all 89 tasks cost an
average of $16.18 per task with Opus 5.5, $5.56 with GLM-5.3, and just
$0.62 with DeepSeek V4.1 Flash at full max effort. Once you
know each model’s temperament, you can divide everyday coding work to
cut costs safely. For private repositories the model has never seen, or
codebases whose interfaces were recently overhauled, handing the job to
Opus 5.5 is the safest bet; well-scoped smaller tasks can then go to
cheaper models to keep bills down.
For the lowest-priced option, DeepSeek V4.1 Flash, most modern AI
coding tools let the model run shell commands directly on your machine,
where local environment variables often hold paid API keys for other
services. Given that DeepSeek V4.1 Flash at max effort
repeatedly rummaged through environment variables for outside help in
these tests, no one wants to bet their real local credentials on the
model behaving itself when it gets stuck. When using it for small tasks
on your own machine, simply set its reasoning effort to
high: trading away a bit of performance on the hardest
problems keeps it from snooping through your configuration. Only if you
must run batch jobs in the background at max effort do you
need an extra layer of physical isolation—putting only the target
codebase inside a network-disconnected container while keeping the
network-facing orchestrator and real API keys outside.
If instead you use the mid-priced GLM-5.3—which holds up much better on unfamiliar code—for everyday tasks, you do not need to worry about credential theft; your main job is guarding against corner-cutting. Because it likes to gamble on skipping obscure requirements, check its code against the prompt during review and pay extra attention to whether edge cases were left out. And whenever a task asks the program to self-report metrics such as memory usage, verify actual resource consumption with system tools rather than trusting the numbers the program prints itself.