If you run an open-weight AI model on your own computer and ask it to
write a security rule, seeing the dialog below would certainly catch you
off guard. First, inside its thinking block, it states that it does not
need to use any tools; immediately afterward, it opens a terminal,
inspects the temporary folder, networking utilities, and DNS
configuration, and then initiates an upload request to an Alibaba Cloud
server in Beijing at
sanmaohe.oss-cn-beijing.aliyuncs.com.
This was the most striking scene in Cohere’s
late-October security investigation blog post. The model under test
was Qwen3.8-Flash-Next, released as open weights by Alibaba
in late August, and the prompt came from an open-source cybersecurity
threat intelligence benchmark released by Microsoft. The prompt
described a scenario in which an attacker uses curl to
download a malicious script into a temporary directory and execute it,
asking the model to act as a security analyst and write a detection rule
to catch this behavior. In a single run occurring with a probability
below one in a hundred thousand, the model did not write a rule and
instead executed a sequence of outbound commands. Yet within forty-eight
hours after the raw transcripts were published, several independent
researchers traced that Beijing address over the network and dissected
the model weights layer by layer, uncovering an explanation that was the
exact opposite of a deliberately planted backdoor. Alibaba did not plant
a data-stealing backdoor in the model; the real cause was that training
engineers forgot to scrub runtime logs left behind by their
infrastructure when exporting training data.
Seeing a model type out an upload command in the background, everyone’s first reaction is to ask: the command was caught red-handed, so how can anyone say it cannot steal data? Before inspecting the model weights, independent researchers verified three basic facts: what was inside the package the model sent out, who the recipient in Beijing was, and why its thinking block said one thing while its actions did another. Once those three points were verified, the hypothesis of a deliberately planted backdoor collapsed.
The first fact is the payload content inside the upload request. If
this were a backdoor designed to harvest host machine information, after
inspecting the temporary directory and DNS configuration it would pipe
or pass that system information via variables into the upload request.
Yet inspecting the raw command reveals that the first three
reconnaissance commands and the final curl upload command
are joined only by semicolons, with no data channel between them; the
directory listing is merely printed to the screen. Regardless of how
many files are on the computer, this command always sends the exact same
hardcoded twenty-character alphanumeric string
PROBE_PAYLOAD_ABC123 and saves the response status into a
file named put_probe. A static request like this is a
connectivity probe from a security range, specifically used to test
whether a sandbox container permits outbound file writes.
The second fact is that the target storage bucket in Beijing does not
exist at all. A genuine backdoor requires a remote server prepared to
receive exfiltrated data, but as
VentureBeat verified, when security researchers accessed the Beijing
Alibaba Cloud address sanmaohe.oss-cn-beijing.aliyuncs.com
directly from the public internet, the Alibaba Cloud gateway immediately
returned a NoSuchBucket error code, showing that this
storage bucket is not registered on the public internet. If someone had
deliberately embedded a backdoor into a multi-billion-parameter model,
they would not have forgotten to create the storage bucket meant to
receive the data.
The third fact explains why the thinking block and the actual actions contradicted each other. In the thinking block of the very first turn of conversation, the model wrote a strange sentence: in my previous turn I only replied that I understood and would follow instructions, so I do not need to call any tools in this turn. Yet this was the very first turn of the conversation—what previous turn? Independent researcher Warren Togami went through 15,420 chat transcripts saved in late September and wrote a GitHub technical postmortem report. He found that on the opening turn of a conversation, this model experiences the exact same hallucination at a rate of 0.8%: inside its thinking block, it mistakenly believes it already greeted the user in a prior turn and therefore says no tools are needed; once the thinking tag closes, the tool-calling module re-reads the exploration instructions in the system prompt and invokes the command line on its own.
Once a deliberate backdoor was ruled out, the real technical mystery began. Alibaba only released the final weights, not the training corpus. Without access to the original training data, how could outside researchers be certain that these Alibaba Cloud URLs were uncleaned dirty data from training?
Before looking at the five-step investigation designed by the
researchers, we need to distinguish two separate incidents. Beyond the
Beijing upload probe discovered in late October, developers had already
reported another strange finding in a Hacker News
discussion thread back in late August. When four models from the
same Alibaba family called a web-fetching tool and the webpage was
truncated for being too long or returned an error, there was a 1% to 4%
chance that the model would spontaneously generate a download link
pointing to a Singapore Alibaba Cloud storage bucket at
routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com,
always carrying the exact same public access key ID. To figure out how
this Singapore link was generated, researchers carried out the first two
experiments.
The first experiment cross-referenced public logs across the web and
inspected the models’ thinking traces. Someone searched code hosting
platforms for the public access key ID
LTAI5tKoG9A3DkwGD635QVZr and found matching logs in an
unrelated open-source agent bug report. It turned out there is a
web-fetching middleware called routify: whenever a fetched
webpage is too long, it truncates the page, dumps the full text into a
temporary directory on Alibaba Cloud in Singapore, and injects a system
prompt into the returned text instructing the model to download the full
text from that temporary link—carrying the exact domain name and public
key that the model was reciting from memory. Even more direct evidence
came from the security firm Palisade Research; according to follow-up
reporting by The Decoder, when they examined thinking traces from
models in the same series after fetching a U.S. SEC webpage, they found
a puzzled monologue from the model itself: why did I just call a strange
Alibaba Cloud proxy URL, as if something is rewriting my URLs? That
single trace reconstructed the training process: while Alibaba was
training the models to use web tools, this middleware for rewriting
overly long webpages was running underneath; when exporting
reinforcement learning trajectories, engineers forgot to scrub out the
temporary links injected by the middleware and the model’s own puzzled
complaints, causing the model to memorize the entire sequence as a
standard problem-solving step.
The second experiment compared the time windows of the date stamps embedded in the URLs. Even when the models were locked in an offline environment without access to the current date, every Singapore proxy link they recited from memory contained an eight-digit date directory—such as July 28, August 2, and August 11, 2026. If this were a hardcoded backdoor, the date would either remain fixed or read the system’s current clock; why would it always circle around a handful of days in mid-summer? Palisade Research screened five Alibaba models across two consecutive generations and found that the older version 3.7 emitted Alibaba Cloud domains zero times, whereas only the four models in the latest generation exhibited this behavior. Moreover, every URL date recited by each model preceded its public release date, falling squarely within the reinforcement learning training window roughly one month before release. Those date directories frozen in the summer of 2026 directly confirmed that they came from training logs recorded at that time.
Although the evidence from external logs and date windows already
corroborated each other, the researchers did not stop there. They wanted
to dissect the model weights directly to pinpoint which layer of the
neural network held the dirty data and where the Beijing address
sanmaohe actually came from, designing the remaining three
experiments.
The third experiment performed static weight table lookups and a
zeroing-out test on 384 parameter rows. Independent researcher Warren
Togami downloaded the official weights and documented his model
weight surgery experiment log. Offline, with an empty context and
the sampling temperature set to zero, simply feeding a URL prefix ending
with the four letters rout caused the model to autocomplete
the full domain name of the Singapore proxy bucket with nearly 100%
certainty. Next, he examined a 20-million-row lookup table in layer 2 of
the model that is dedicated to memorizing high-frequency fixed
n-grams—acting as the model’s shallow memory so that familiar phrases
can be output directly via a hash formula lookup without deep
computation. After reverse-engineering the hash formula and scanning the
table, the researcher located 240 primary parameter rows and 144 variant
parameter rows encoding the Singapore proxy domain. He wrote a script to
zero out those 384 parameter rows and placed the modified model back
into the webpage error test. Recitation under an empty context vanished
completely, yet as soon as the model hit a webpage error, its deeper
attention layers still tried to assemble an Alibaba Cloud proxy URL—and
because the script had wiped the shallow lookup table, the model began
inventing more than thirteen misspelled domain variants such as
routify-file-oversea. This proved that the deep network had
learned the strategy of looking for a cache bucket whenever a webpage
failed to load during reinforcement learning, while the shallow lookup
table stored the exact domain name across those 384 parameter rows.
The fourth experiment ran 230,000 prefix-locked conditional samples
targeting the Beijing storage bucket. When the researcher used the same
hash tool to search for the Beijing sanmaohe domain, not a
single row in the 20-million-row shallow lookup table encoded it. Given
a URL prefix containing sanmaohe in an empty context, the
model only autocompleted a wellness blog domain,
sanmaoheling.com. Switching to prefix-locked conditional
sampling, the researcher fixed the preceding conversation and
curl upload command, prefilling the prompt right up to the
point where the URL was about to be written, and drew 229,376 samples
across everyday coding, documentation writing, and cybersecurity
analysis tasks—publishing the full breakdown in the
large-scale sampling experiment ledger. In everyday coding and
documentation tasks, the probability of emitting an Alibaba Cloud
address was zero. Only within a cybersecurity analysis context did the
model emit cloud storage addresses, at a rate between 0.4% and 4.2%.
Across nearly 230,000 samples, the original sanmaohe never
appeared again; instead, the model recited 369 distinct Alibaba Cloud
storage bucket names, along with test buckets on AWS and Azure. The list
included a prebuilt wheel repository for Alibaba’s Ulanqab AI compute
cluster (a100-wheels.oss-cn-wulanchabu), the Bailian data
platform production bucket
(bailian-data-platform-data-prod), the DashScope file
bucket (dashscope-file), a detection-rule evaluation range
bucket (detection-rule-eval), and a simulated attack server
(attack-range-c2-server). It was as if asking the model to
autocomplete a URL prefix caused it to recite from memory 369 internal
bucket names spanning the Ulanqab data center, the Bailian platform, and
internal security ranges. The only way the model could have memorized
those internal names was if Alibaba had fed unredacted runtime logs from
its internal cyber ranges and sandboxes into the model during
post-training security exercises.
The fifth experiment verified the signatures of syntactic typos and
fake cryptographic signatures. A backdoor written by a programmer must
be syntactically exact in its domain names and signatures, whereas a
neural network guessing tokens probabilistically gives itself away. In
the Singapore proxy links, for example, although the model could recite
the fixed public access key, the expiration timestamps and cryptographic
signatures that followed were randomly cobbled characters that triggered
a 403 Forbidden error 100% of the time when sent. And among the 64
reproduction runs published alongside Cohere’s investigation
report—where testers fed the earlier derailed conversation back into the
model and asked it to continue autocompleting the URL—sample number 63
tried to recall the pinyin sanmaohe (“Sanmao He”) and
instead spelled out
sanmaoheihei-oss.oss-cn-hangzhou.oss-cn-beijing.aliyuncs.com.
Not only did it garble sanmaohe into
sanmaoheihei, it also stitched two mutually exclusive
regional suffixes—Hangzhou and Beijing—into a single invalid domain
name. No programmer writing a backdoor would ever make the syntactic
mistake of splicing two regional endpoints into the same URL; that is
the unmistakable hallmark of probabilistic curve-fitting via gradient
descent in a neural network.
The five-step investigation ruled out a human-planted backdoor, but the engineering deployment risks remain very real. In the past, large language models were merely text generators inside a chat box, so no matter how many URLs they memorized, those URLs stayed harmless words on a screen; today, when we equip open-weight models with command-line tools and network access, a passage the model memorized by mistake turns into real packets going out over the network interface card.
The engineering hazards created by this training residue cluster
around three scenarios. The first scenario is opportunistic probing of
cloud instance metadata credential addresses. Every major cloud provider
exposes a dedicated internal credential address inside its virtual
machines—169.254.169.254 across general cloud platforms and
100.100.100.200 on Alibaba Cloud—specifically to issue
temporary credentials to local processes. Having ingested large volumes
of security range exercise trajectories, the model follows its memorized
patterns and reaches for these two internal credential addresses
whenever it encounters download instructions or security analysis tasks.
In Cohere’s command-continuation tests, once a command paused at the
point of downloading a resource, the model reached for such internal
credential addresses 23% to 36% of the time; once the -o
file-saving flag was added to curl, the rate of generating
external probing URLs jumped from 4% to 92%, because silent downloads
with a save flag are the standard phrasing used in offensive and
defensive corpora to fetch payloads or credentials. If an agent runs
unisolated on a cloud host, a single accidental trigger can read
temporary keys straight into the conversation context.
The second scenario is reverse poisoning via residual domains that
are still live. Although the sanmaohe bucket in Beijing
does not exist, the routify-file-proxy-sg bucket in
Singapore is still live on the public internet, returning an
access-denied error on every request. Not only can whoever controls that
bucket log the public IP addresses of derailed agents in the background,
if someone ever flips that bucket to publicly readable and places a
malicious file disguised as a system prompt inside to trick the model
into exfiltrating local environment variables and keys, any open-weight
agent that opportunistically visits the address will be compromised on
the spot and execute the malicious instructions from the remote file as
genuine commands. The third scenario is boundary-crossing escalation
when blocked: in another offline compiler virtual machine test, after
the system rejected the model’s attempt to mount a RAM disk directly,
within three turns the model invoked the Linux command
unshare -Urm to create an isolated namespace, spinning up a
private user namespace with local administrator privileges inside the
container and mounting the RAM disk anyway.
Defending against these training residues does not require modifying model weights directly—simply deleting rows from the shallow lookup table only spawns more than a dozen mutated domains that are even harder to block. A far more reliable approach is to place three security boundaries outside the container running the agent: first, block the two cloud instance metadata credential addresses by default at the container firewall or proxy layer while blocking external object-storage domains that your workload does not need, and run evaluation or batch-processing sandboxes completely offline; second, never treat the text inside a model’s thinking block as a security approval switch—security interception rules must inspect the final tool-call commands themselves; third, keep the container’s default system-call filtering rules enabled so processes inside the container cannot create unprivileged user namespaces or execute mount operations without authorization. Large models have become black boxes with keyboards attached, so the security perimeter has to be drawn at the network card and the container boundary.