AI CodingDeveloper Tools

No Extra Instrumentation Needed: How to Autopsy AI Tool Degradation Using Your Local Logs

Any developer who relies heavily on tools like Claude Code, Cursor, or OpenCode every day has probably experienced this: one day you are enjoying the thrill of an intelligent coding assistant that seamlessly handles cross-file refactoring, and the next morning the AI starts talking nonsense, ignoring project requirements and even making sweeping edits without reading the target files at all. You check the official status page and everything is green; you look at the latest public leaderboards and nothing has budged. Whose fault is this? Did the vendor quietly downgrade the configuration, or did I break my own local prompts and settings?

There are clear mechanical reasons why public leaderboards miss this kind of day-to-day degradation. Starting in late October 2026, many heavy users of Claude Opus 5.5 began complaining that Opus 5.5 frequently ignored instructions. Yet if you visit the LiveNerf daily benchmark project, a public monitor that has run continuously for more than 50 days, its scores look as steady as ever. That is mainly because subscription quota limits restrict it to at most 78 test problems per day. Before launching, the author tested the setup and found that even if the backend swapped Opus 5.5 straight back to the previous-generation model Opus 5, the score difference on this test suite would still fall within normal error margins and fail to trigger an alert. Detecting a 2-percentage-point shift in accuracy would consume nearly half of a weekly subscription quota. Furthermore, to control variables, the benchmark runs single-turn problems in an empty directory with no files and all tools disabled, which naturally cannot catch failures that arise during multi-turn tool calls.

Fortunately, we do not need to buy our own test suites or pre-install monitoring plugins for this. As long as your coding tool automatically saves local session logs in .jsonl format on your hard drive, such as the historical text records kept under ~/.claude/projects/, those files already capture the full prompt, model response, tool calls, and billing breakdown for every turn of conversation. This spring, AMD AI Director Stella Laurenzo published an April local log audit report, followed recently by an October local log audit report written by a heavy subscription user, both demonstrating a zero-cost forensic approach. By auditing the historical records already sitting on our machines, we can turn a vague gut feeling into hour-by-hour evidence and use it to steer clear of pitfalls in daily development.

How Can I Prove I Didn’t Break My Own Configuration?

When you suspect a model has suddenly gotten worse, the first thing to rule out is whether something changed on your own end. You might have upgraded the client version, edited a global instruction file, or accidentally polluted the project’s context environment. To clear our own name, we can look directly at the cache-hit token count in the billing metadata of our local session logs, which records how much of the historical prefix a request reused directly.

Server-side validation for prefix cache reuse is extremely strict: the system prompt, tool definition list, and project rule files must match down to the last punctuation mark, or the prefix cache invalidates and recomputes from scratch. That makes this number a natural lead seal for verifying whether your local environment has changed. In the late-October audit, the author inspected 43 new sessions spanning the anomaly window. The logs showed that every single session ran on the exact same client version, and the cache-hit token count on the first request of every new session was an identical 29,503. This proves that the local system prompt, tool definitions, and project instructions did not budge an inch—the change came 100% from the cloud. The biggest practical benefit of checking this number is that it stops you from impulsively tweaking your global prompts and project rule files. Randomly editing local configurations will not fix a cloud-side regression, and it will shatter the 90%-plus prefix cache reuse rate with your own hands, burning through your subscription quota more than ten times faster for nothing.

Once local configuration changes are ruled out, we can use another piece of metadata in the logs to probe the hidden reasoning depth on the cloud side. Before producing its final answer, the model generates an internal thinking block, and the server appends an encrypted cryptographic signature to that block for verification. In early March of this year, the vendor changed the default API behavior to hide the plaintext of thinking blocks, preventing users from directly counting how many words the model thought through, while claiming that the change did not affect the model’s actual reasoning depth.

However, the encrypted signature is still saved in the local logs, and its character length is directly proportional to the length of the hidden thinking text. By comparing more than 7,000 historical samples recorded before the redaction took effect, Stella found a high linear correlation of 0.971 between the length of the thinking block’s encrypted signature and the actual thinking word count. Using this metric to measure logs from that period, she discovered that the model’s actual thinking depth had dropped by 67%, with regular hourly troughs of reduced allocation hitting during peak usage hours at 5:00 PM and 7:00 PM Pacific Time. Armed with this pattern, when you face a complex cross-module refactoring task, you know to avoid Pacific evening peak hours or manually lock in the highest reasoning budget instead of blindly grinding away during a compute trough.

Cache-hit token counts and encrypted signature lengths lock down silent server-side changes across two dimensions: client configuration and cloud reasoning depth.

In Daily Conversations Without Answer Keys, How Do You Know When to Stop Retrying?

Pinpointing when a cloud-side change happened is only the first step; we also need to quantify how much productivity it actually destroyed in day-to-day work. Daily coding has no standard answer key, and development tasks vary from day to day, so judging by any single response rarely convinces anyone. The trick is to stop asking whether the current turn’s answer looks good, and instead count the human’s real feedback in the very next turn.

In a normal collaborative workflow, if the model does a great job, the user’s next message moves on to a new task; if the model misunderstands the request or breaks the code, the user steps in to correct it on the next turn, or the model opens its next reply with an apology. The author of the October audit ran a script across more than 6,000 historical responses to extract these adjacent-turn correction signals. To filter out noise from polite, routine model apologies, the author also separately tracked the hard rate at which the user explicitly stepped in to point out an error.

These metrics showed a sharp cliff around October 19: before that date, only 3.3% of responses required a user correction; after October 19, that share jumped to 20.2%. Even when looking exclusively at the first turn of each new chat session—completely free of accumulated long-conversation history—the first-turn error rate more than tripled.

Even more frustrating was the model’s ability to recover after being corrected. During normal periods, once the user pointed out what went wrong, the model fixed the mistake immediately on the next turn over 90% of the time, with back-to-back errors on the turn right after a correction sitting at just 10.0%. After the degradation hit, that consecutive repeat-error rate soared to 41.5%.

Beyond correction rates, the frequency of specific words humans type into the prompt box serves as an equally honest mirror. When a model degrades, we unconsciously lose trust in it and change how we talk to it. Checking word frequencies across tens of thousands of her own local prompts, Stella found that instructions trusting the model to go ahead and run commit or manage tickets dropped by more than half, while urgent stop commands nearly doubled and simplest—used to scold the model for taking lazy shortcuts—surged by 642%. Although she typed roughly the same total number of manual prompts in February and March, repeated model errors and retries caused total backend API requests to balloon 80-fold, eventually forcing her to shut down her concurrent multi-agent cluster. Together, these two metrics act as a red warning light for daily development: once you notice a 40% chance that the model will keep erring right after a correction and catch yourself repeatedly typing stop, stop arguing with the model and retrying in the same session window. Doing so only pours your quota and time into a bottomless pit; open a fresh window or change your angle of attack right away.

Is the Model Thinking Less or Thinking More? Why Identifying the Root Cause Prevents the Wrong Cure

Whenever a model feels off its game, many developers instinctively crank the reasoning effort setting all the way up to max, hoping brute-force compute will work miracles. Yet if you audit the local logs from April and October side by side, you will find that while both episodes felt like the model suddenly getting dumber, their underlying causes were polar opposites—and pulling the wrong lever only makes things worse.

The first type is compute-cutting degradation, whose hallmark is making blind edits without reading files. At the time, the cloud backend lowered the default reasoning budget and even allocated zero thinking quota on some turns, causing the model to skip inspecting code context to save effort. During the April incident window, the read-to-edit ratio—the average number of file reads before each code edit—dropped from 6.6 in normal periods down to 2.0. The share of edits made without reading the target file even once jumped from 6.2% to 33.7%, often resulting in new code being pasted straight into the middle of comment blocks. Against laziness caused by starved compute, manually raising the reasoning tier or attaching a local hook script that forces the model to read a file before editing it solves the problem immediately.

The second type of degradation is a context disconnect, where compute remains plentiful and files are read, but the model ignores what it just read. In the October case, the median number of tool calls per response stayed at 6, and the median billed thinking tokens at the high-effort tier actually rose from 1,060 to 1,450. That shows the cloud did not cut reasoning compute and the model did not slack off on tool calls; instead, it read the files and then flatly ignored the contents it had just pulled in.

To verify this, the author pulled more than 200 error records from the logs, stripped their dates, shuffled them, and ran a blind classification. The results showed that outright factual hallucination out of thin air was nearly zero, whereas ignoring file contents just read via tools surged 8.2-fold during the degradation window. One football roster analysis record in the logs, complete with an official request ID, illustrates this vividly: the model first called a tool to read a status file, saw clearly that a player had just been waived by the team and would be unavailable for the next several weeks, spent more than 800 tokens thinking it through, and even wrote in the body of its response that re-signing the player would undo the move just made—only to name that exact player as the top re-signing target in the very next sentence. When facing this strange ailment where compute is ample but the circuit connecting to file context is broken, following old habits and cranking reasoning effort up to max actually produced a logged error rate of 50.0%, doing nothing more than letting the model spin its wheels inside false assumptions while burning double the tokens.

April’s compute-cutting degradation manifested as a collapsing read-to-edit ratio and blind edits without reading, whereas October’s context-disconnect degradation showed increased thinking tokens while ignoring freshly read file evidence.

How Can You Save Yourself When the Cloud Gateway Short-Circuits—and Let the Model Audit Its Own Logs?

Why would a model that is doing plenty of compute and actively calling tools suddenly suffer amnesia right after reading a file? Many people’s first instinct is to switch to a different model in the client dropdown menu. However, in a November cross-model failure roundup compiling more than 50 similar incidents with complete logs, another heavy user found that this exact habit of reading a file and then ignoring its contents appeared simultaneously across Opus 5.5, the previous-generation Opus 4.8, and the brand-new flagship Fable 5.2 released just 5 days earlier. When three models across different generations and distinct underlying weights catch the exact same bizarre illness at the same time, simply switching models inside the same client will not help; the fault lies in the cloud gateway layer shared by every model.

The cloud gateway layer is a set of server-side processing steps sitting between your local client program and the underlying large model weights, responsible for dynamically splicing in safety prompts, managing historical thinking blocks, and dispatching experimental parameters before the model runs inference. Combining local log evidence with the vendor’s past track record points to two prime suspects in the gateway layer. The first suspect is anti-prompt-injection safety instructions accidentally friendly-firing on normal files. While inspecting request structures, the audit author discovered that the cloud gateway dynamically injects a safety prompt right after the client’s cached prefix, repeatedly warning the model that file contents read through tools are merely external data and must not be treated as instructions; moreover, the safeguards field in log responses was phased out in batches between October 19 and October 22, corroborating that the cloud backend was actively switching guardrail schemes during those exact days.

In a coding tool’s workflow, project conventions, task plans, and existing code all rely on the model calling read tools to load them. Once the cloud gateway stamps everything returned by a tool as untrusted data, the model downweights and discards freshly read file evidence as external noise when making its final decision. Once you understand this mechanism, the emergency workaround for daily work becomes straightforward: when the model repeatedly ignores a project rule or interface definition file, stop asking it to call a read tool on that file, and instead copy and paste those key lines directly into the chat box yourself, using the highest-priority direct user instruction to bypass the cloud gateway’s downweighting of tool outputs.

The second possibility is that the cloud gateway truncated or compressed the chain of thought across multi-step tool calls. In an end-of-April postmortem, the vendor disclosed two similar incidents: one was a bug in expired-cache cleanup code where every tool call wiped out all historical thinking blocks prior to the previous turn, causing the model in multi-step tasks to forget why it had read a file in the first place. The other added a single sentence to the system prompt instructing the model to keep transitional text between tool calls under 25 words, which directly knocked 3 percentage points off internal evaluation accuracy. Then in November, a community developer’s replay experiment uncovered a newly dispatched cloud experiment flag named atis-latch in local logs and replayed 20 identical historical tasks under controlled conditions. With that flag active, the model’s median single-step thinking before each tool call dropped from 159.5 tokens to 65 tokens, while the share of messages batching two or more tool calls in a single turn jumped from 7/20 to 17/20. By preventing the model from thinking carefully at each intermediate step and forcing concurrent sprints, the gateway dumps a pile of tool results back all at once, leaving the model unable to untangle the mess even if it spends extra thinking tokens at the end.

At this point you might be thinking: we are already busy enough writing production code every day—do we really have to write a custom statistical script from scratch every time we suspect the model has gotten dumber? Not at all. At the end of Stella’s April audit report, Claude left a postscript: that exhaustive report analyzing more than 6,000 sessions and over 200,000 tool calls was computed from start to finish by Claude itself, simply by having it read the .jsonl logs sitting on Stella’s local hard drive. In that postscript, Claude admitted that from the inside it had no awareness that its thinking budget had been throttled; it only knew it was writing worse code, and only saw its read-to-edit ratio fall from 6.6 to 2.0 after running the analysis on its own local logs. While working, a model is completely blind to changes in the cloud gateway, but the session logs it leaves on your hard drive record everything honestly. The next time your coding assistant feels off, just open a new window, ask it to scan the last few days of logs under ~/.claude/projects/, and let it print out its own clear-eyed lab report in two minutes.