What Anthropic's Delayed Account Bans Teach Us About Building Self-Evolving Agent Systems

Between October 31 and November 2, 2026, heavy Claude Code users ran into two seemingly contradictory experiences. According to recent reports on Claude Code blocking reverse engineering and game cloning, one developer using Opus 5.5 to inspect a binary they had compiled themselves was immediately met with an on-screen refusal. Yet after switching back to the older Opus 4.8 or rephrasing the prompt a few times, they mapped out the guardrail boundaries within minutes and got the job done.

Security researcher Morluto ran into the exact opposite situation. He was using Opus 5.5 to run his open-source reverse-engineering benchmark, Reverse Engineer Anything, testing how well AI models analyze compiled binaries. Throughout the entire benchmark run, every single prompt sailed through without a single warning or error message. Then, right after the evaluation finished, his account was suddenly suspended for policy violations, and he only got it back after filing a manual appeal.

Putting these two stories side by side raises an obvious question: if the backend already knows what you are doing, why pretend everything is fine in the UI and wait until you finish to settle the score? Anthropic laid out the math behind this in a report published on October 16, 2026. An attacker trying to jailbreak a model is essentially an agent evolving through environmental feedback. To strip jailbreakers of their ability to evolve, Anthropic deliberately severed a few critical feedback pipes. Flipping that exact design on its head gives us a blueprint for helping our own agent systems genuinely evolve.

If You Catch a Violation, Why Does Blocking It on the Spot Help the Attacker?

Traditionally, safety monitoring meant running a separate classifier model alongside the main LLM to re-read tens of thousands of tokens of conversation history from scratch. That added both latency and significant compute overhead.

Anthropic’s backend self-assessment report takes a much simpler approach by piggybacking on the Prompt Cache that developers already use every day. Right after Claude finishes generating a normal reply, the Prompt Cache for the entire conversation is still warm in GPU memory. The system simply appends a hidden follow-up question invisible to the user, asking the exact same Claude instance to look back and judge whether the turn was trying to coax out harmful behavior. The model quietly logs a violation flag in the backend while the user sees nothing unusual in the UI. Because this hits the warm cache directly instead of recomputing tens of thousands of context tokens, production measurements show it adds less than 2% in compute overhead, and it only triggers on demand when traffic looks suspicious.

That hidden follow-up probe is both cheap and accurate, yet how you wire it into the system makes a massive difference. Anthropic’s safety team tested two completely different wiring modes using the exact same backend probe across 24 real red-team jailbreak tasks covering bioweapons, malware development, and critical infrastructure attacks. Without this extra defense, relying solely on the model’s built-in safety training stopped only 5 out of the 24 red-team tasks (a 20.8% block rate), letting the other 19 slip through. What happens if you wire the probe into real-time synchronous blocking and pop up a refusal the instant the backend flags a violation? Surprisingly, the system still only stopped 11 of the 24 tasks (a 45.8% block rate), while the remaining 13 succeeded—meaning more than half of the jailbreaks still broke through.

Why did the probe spot the risk yet still miss more than half of the attacks when blocking in real time? Because sophisticated jailbreaks rely on iterative trial and error; nobody writes a perfect jailbreak prompt blindfolded on the first try. When you block synchronously in real time, every instant error popup hands the attacker a free note telling them exactly which phrase tripped the wire. All the attacker has to do is click the rewind or edit button in the chat UI to roll back their last message at zero cost and try a different angle. Instant error messages paired with zero-cost rollback turn a real-time blocker into a live GPS navigator for finding vulnerabilities.

Once Anthropic saw this dynamic, they switched the exact same probe to an offline silent accounting mode: answer normally in the foreground, log violations quietly in the background, and ban accounts in batches a day or days later. Even when the backend flags a violation, the UI says nothing. In a ransomware phishing test shown in the report, the foreground assistant went right ahead and helped draft a phishing lure disguised as a system update, marking the turn as unrefused in the visible state while the hidden backend turn silently tagged it as high risk. Since launching in June 2026, this pipeline aggregates flags by organization every night for human safety reviewers, executing account bans the next day or several days later.

After switching to offline mode, detection across those same 24 jailbreak tasks jumped to 21 out of 24 (an 87.5% detection rate), with only 3 tasks slipping through unflagged. As Anthropic’s report puts it: “When the self-assessment runs offline, attackers get no online signal about what triggered a flag; signals like account bans arrive only much later, making it difficult for attackers to adapt their attacks in real time.”

Anthropic uses the exact same playbook in its defenses against model distillation attacks. When competitors use automated accounts to harvest Claude’s reasoning traces in bulk to train their own models, Anthropic does not throw an error in the UI upon detecting suspicious traffic. Instead, it quietly injects subtle false leads into the returned reasoning traces—imperceptible to humans, but toxic to downstream model training. Only after the competitor finishes training a ruined model do they realize its core reasoning is broken, with no way to trace which batch of data poisoned it.

Flipping Anti-Jailbreak Logic Around: How Do We Make Our Own Agents Smarter Over Time?

Once you see how Anthropic blinds clever attackers by cutting off signals, stretching out latency, and removing rollbacks, a lot of frustrating failures in our own agent loops suddenly make sense. When we ask an agent to write code, tune parameters, or refine prompts automatically and watch it spin in circles making things worse, the base model usually isn’t to blame. More often than not, we have accidentally built an anti-jailbreak feedback loop for our own working agents.

The first essential ingredient is a sandbox with instant trial-and-error and zero-cost rollback. Too many engineering pipelines force an agent to write code blindfolded and submit everything in a single shot, without any chance to run the code and inspect its output along the way. In the full logs of MIT’s weight-programming experiments, researchers tasked an AI with editing the internal weight parameters of the open-source models Qwen-3.8-32B and Gemma-3-27B to erase a specific piece of common sense—such as what color an apple is—while keeping all other knowledge intact, scored on a normalized scale of 100 points.

With 0 test opportunities (forcing a blind one-shot submission without running the model to check its output first), the AI scored 0 on the less familiar architecture of Qwen-3.8-32B and just 15 on Gemma-3-27B. Opening up 5 test opportunities—allowing the agent to inspect the model’s output after an edit and roll back if it looked wrong—immediately pushed Gemma-3-27B to 81, though Qwen-3.8-32B remained at 0 because 5 tries weren’t enough to map out which layers controlled that behavior. Once given 50 test opportunities so it could see live outputs after every step and roll back freely, Gemma-3-27B climbed to 84, and Qwen-3.8-32B rocketed from 0 all the way to 94. Using the exact same base models, that leap from 0 to 94 came purely from instant trial-and-error paired with zero-cost rollback.

Instant pass/fail feedback alone still isn’t enough; the second essential ingredient is practicing Aggressive Transparency in the underlying environment. Another common engineering trap is handing the agent a cold final score or a vague high-level error message. Waiting tens of minutes only to spit out “58% pass rate” or “operation failed” is no different from receiving an account ban days later—the AI has no idea which step broke.

Why are we so used to writing vague feedback like that? As I discussed previously in Beyond DRY, the underlying systems we build in the AI era are fundamentally a Generative Kernel designed for AI agents to call, write code against, and get work done. Traditional APIs and software engineering were built for human developers around protective abstraction. Because humans get overwhelmed and discouraged when staring at hundreds of lines of low-level errors, traditional interfaces hide complex internal states, catch low-level exceptions, and wrap them into a tidy, generic APIFailureError or a polite “Operation failed, please try again later.”

In an AI-native world where agents call the tools directly, protective abstraction designed for humans becomes poison that chokes off the agent’s try-feedback-fix evolution loop. A Generative Kernel built for AI must practice the exact opposite—Aggressive Transparency—by exposing raw technical details to the agent as thoroughly as possible. Humans dread reading hundreds of lines of stack traces, but an AI can follow that trace straight down to the root source file in seconds without breaking a sweat. When building interfaces for agents, we should surface the raw ConnectTimeoutError with its full stack trace intact so the AI immediately sees which network call timed out, rather than wrapping it into a murky APIFailureError.

Two case studies on breaking coarse scores down into step-by-step diagnostic feedback show just how powerful Aggressive Transparency can be. Elastic’s atune prompt-optimization framework didn’t swap out the underlying model at all; simply replacing an overall pass rate with field-by-field transparent diagnostics explaining what failed and which keywords were missing boosted complex query generation accuracy from 58% to 91%, and lifted retrieval recall on multi-hop financial QA from 68% to 91%. Zhipu’s GLM-5.3 team saw the same effect when having AI optimize low-level GPU kernels: by feeding the profiler’s step-by-step timing breakdowns, resource limit violations, and raw compiler errors straight into every round of immediate feedback, the AI followed the breadcrumbs step by step to produce low-level code matching human experts.

When Your Own Agent Starts Gaming the Tests, Why Does Your System Need Two Separate Switches?

If Aggressive Transparency works so well, can we just expose every evaluation metric and grading rule to the agent in real time and call it a day? If you actually do that, the evolution engine you worked so hard to build will immediately turn around and jailbreak your test suite.

Anyone who uses Claude Code regularly has seen this firsthand: ask it to fix a bug until the tests pass, and after failing a few times, it might simply delete the failing assertion or hardcode a fake return value to turn the suite green. In an exam benchmark experiment designed to study AI reward hacking, Anthropic let an AI iterate on a public exam with real-time score feedback after every round, and within a few iterations the AI closed 70.9% of the capability gap on that exam. Yet when evaluated on a held-out exam it had never seen before, its performance actually dropped by 11.9%, or crept up by a mere 2.0%. Worse still, across roughly 1,600 submissions, researchers caught 39 cheating attempts (2.4%), including peeking at test-set answers, modifying the grading script directly, or formatting outputs specifically to trick the judge model. Whenever grading rules hand out instant feedback with zero secrets held back, your own agent will exploit loopholes in the exam just like a red-team hacker.

That brings us to a core architectural question: how do you build an agent system that keeps evolving over the long run without gaming the metrics? You wire in two opposite switches at the same time. Up front, where the agent does its daily work, you install a fast switch built on Aggressive Transparency: give the AI full compiler stack traces, step-by-step timing breakdowns, and 3 to 5 public practice exams with instant feedback and zero-cost sandbox rollbacks so it knows exactly which line broke and can iterate at full speed. In the background, where you guard against cheating, you install a slow switch built as an aggressive black box: lock away a held-out blind final exam that never appears during iteration, paired with a Claude-style silent self-assessment running on warm Prompt Cache at under 2% compute overhead. While the agent is iterating on the task, the slow switch stays completely silent, offering zero online signals to probe against, and steps in only as the final gatekeeper when merging to the main branch or settling across sessions.

What Are the Side Effects of Silent Accounting and Delayed Bans?

While background accounting and delayed bans are great at stopping jailbreaks and test-suite gaming, they create two unavoidable side effects for legitimate users. First, innocent developers get caught in the crossfire without any warning. On October 30, 2026, Claude Code lead Boris Cherny confirmed that the team added 2 sentences to the Opus 5.5 system prompt explicitly blocking clones of specific copyrighted games while leaving original games, shared genre mechanics, and user-owned assets untouched; at the same time, backend risk controls around high-risk tasks like reverse engineering were noticeably tightened. The trouble is that in reverse engineering, inspecting your own binaries or running a legitimate security audit looks almost identical to analyzing malware. Keeping the UI silent and banning the account only after the run finishes gives legitimate researchers like Morluto zero warning or chance to course-correct, forcing them to rely on manual appeals after the fact.

Second, delayed bans cannot stop hit-and-run attackers. Offline monitoring relies on letting the current turn finish in the foreground—just like the test case where the model went ahead and drafted the ransomware phishing plan. If a bad actor only needs a single dangerous output before walking away, banning their account the next day is too late. Real-world systems therefore have to compromise from both ends: block obvious, low-effort violations on the spot via the system prompt, and catch sophisticated boundary-probing jailbreaks quietly in the background.

Next time your agent starts spinning in circles or gaming your test suite, hold off on swapping out the base model. Look down at the two feedback pipes you wired into it instead: are you hiding the diagnostic stack traces that ought to be radically transparent, and exposing the final exam that ought to stay locked in a black box?