Anyone who has used Claude Code to refactor complex code across dozens of turns, accumulating hundreds of thousands of tokens along the way, has likely run into the same friction. Even with Prompt Caching enabled to reuse prior computations stored in GPU memory, typing a single word like “continue” still leaves you waiting several seconds before the first word appears. To address this, Anthropic launched the Message Threads incremental transport protocol on November 2, 2026, enabling it by default in Claude Code.
The update shrinks median per-turn request payloads in long conversations from roughly 800 KB down to 1.8 KB, noticeably reducing Time to First Token (TTFT). Yet Anthropic explicitly notes that the model still reads the full conversation history on every turn, leaving token consumption and API bills completely unchanged.
Coincidentally, just one day later, NVIDIA Research published a contrarian empirical study showing that stretching active context to 256K tokens drives the GPU hardware cost per generated token up to 7x higher, advising developers to compact older history regularly. As conversation histories keep growing thicker, why did these two industry heavyweights move in opposite directions in the very same week?
If precomputed prefix caches already sit in GPU memory, why does sending a one-word reply still feel sluggish? Because Prompt Caching only saves matrix computation time after a request reaches the GPU; it does nothing for the API gateway standing outside the GPU cluster.
Under a traditional stateless API, the gateway has amnesia on every request. Deep into a session, the client must re-upload all prior message text, system prompts, and tool definitions on every turn. For computer-use tasks that capture a screenshot at every step, StackOne’s CTO calculated that running dozens of consecutive steps over a stateless API requires uploading 191 MB cumulatively, compared to just 7.5 MB with incremental transport.
A less obvious source of latency kicks in after the server receives the payload. As Claude’s official API documentation explains, once a request hits the gateway, the API must re-validate the schema of every historical message block from scratch on each turn. Because GPUs operate on numeric token IDs rather than raw text, the gateway also has to run hundreds of thousands of tokens through the tokenizer all over again before matching them against the prefix cache in GPU memory. Eliminating redundant uploads, schema validation, and re-tokenization at the gateway cuts median TTFT by roughly 35% on requests of 500k–1M tokens.
Message Threads solves this by adding an encrypted in-memory cache at the gateway layer with a 5-minute TTL. After the initial request sends the full history and receives a response ID, subsequent turns pass only the previous message ID and the newly added turn; re-sending system prompts or tool definitions is not even permitted. If a user steps away long enough for the gateway cache to expire, the next request receives a 404 status code, prompting the client to resend a full request using the complete history saved locally.
If server-side state works so well, why didn’t Anthropic store chat histories permanently in a database instead of building an append-only temporary list that expires after 5 minutes of inactivity? From a deployment standpoint, this lightweight design fits the engineering realities of multi-cloud distribution.
From day one, Message Threads launched simultaneously across Anthropic’s own platform, AWS, Microsoft Cloud, and Google Cloud. Under a lightweight gateway-memory protocol where the client keeps a complete local copy ready for replay whenever needed, every cloud platform can integrate the feature without relying on cross-cloud persistent storage. In fact, the server response does not even tell you whether a given turn was stored; the client simply waits until a subsequent turn returns a 404 status code, then resends the full local draft.
This lightweight protocol also stipulates that the message list in gateway memory can only append new turns to the end and cannot modify earlier turns. On one hand, continuation requests pass only the previous response ID and the current turn’s new messages; the server implements no endpoints to query, edit, or delete older messages, simply retrieving the in-memory snapshot by ID and appending the new messages to the tail. On the other hand, the gateway saves repeated validation and tokenization time precisely because the processed context prefix from dozens of prior turns is already cached in memory; if earlier content needs to change, the protocol requires editing the draft on the client and sending a full request to create a new thread.
This append-only rule clashes directly with four existing mechanisms
in Anthropic’s stack. First, it makes it easy to exceed the limit on
manual Prompt Caching breakpoints. Although the gateway caches chat
history, GPUs do not automatically enable Prompt Caching for you, so
developers still have to manually mark where the GPU cache should end.
The traditional pattern attaches a cache_control breakpoint
to the newest message on each turn; because the API caps a single
request at 4 such breakpoints, developers strip breakpoints from earlier
turns whenever they resend the full history, leaving only the latest
one. Once you switch to incremental threads, previously sent messages
can no longer be edited retroactively. If the initial system prompt and
tool definitions already occupy several breakpoints, or if a 5-minute
timeout triggers a full replay of the local draft and your code forgets
to strip old breakpoints left over from earlier turns, exceeding 4
breakpoints in a single request immediately returns a 400 error.
Avoiding this issue requires using the global automatic caching
parameter so the server automatically advances the cache position to the
final turn.
Second, Message Threads is mutually exclusive with Compaction and Context Editing. Whether condensing conversation history into a text summary via Compaction or automatically pruning older tool outputs and thinking blocks via Context Editing, both features fundamentally require rewriting the message history list on the server. That directly conflicts with an append-only in-memory list, so enabling them together immediately triggers a 400 error; even if the model triggers a refusal on a given turn and that message needs to be deleted, you must abandon the current thread and re-upload the full history.
Third, performance degrades when primary model congestion on the server triggers automatic fallback to a backup model. Each thread is bound to a single model, and when congestion shifts traffic to a fallback model, the fallback model’s responses are never stored in thread memory. Yet to reuse the fallback model’s cache, Sticky Routing on the server may route subsequent thread-creation requests directly to that fallback model for roughly 1 hour, and every response refreshes that 1-hour timer, causing the official SDK to potentially degrade into re-uploading the full conversation history on every turn throughout that window.
Fourth, it crosses Zero Data Retention (ZDR) red lines for financial and healthcare customers. Message Threads stores encrypted conversation content, system prompts, and tool definitions in server memory across requests, expiring after 5 minutes of inactivity and purging completely within at most 1 hour. Because data persists in server memory across independent requests, Anthropic explicitly excludes this feature from Zero Data Retention (ZDR) and HIPAA eligibility.
By contrast, OpenAI takes a different connection model with WebSocket mode in the Responses API. Instead of disconnecting and reconnecting on every turn, the client establishes a persistent WebSocket channel with the server that stays open for up to 60 minutes. Across dozens of consecutive tool calls, the most recent response state lives in the active connection’s local memory cache, so each turn sends only new inputs and overall execution speed improves by up to roughly 40% on workflows with 20 or more tool calls. Server-side automatic Compaction can also be triggered directly inside the connection without re-uploading history afterward. With non-persistent storage enabled, the memory cache is destroyed the instant the connection closes and never written to persistent storage, making it directly compatible with Zero Data Retention (ZDR) requirements.
If long context is so unwieldy, and NVIDIA Research just demonstrated that compacting the window frequently is both faster and cheaper, why haven’t teams building real-world agents compacted every few turns instead of needing Anthropic’s long-context incremental transport? First, we need to work through the GPU physics in NVIDIA’s paper.
Prompt Caching does avoid precomputation during the input phase, yet during the decode phase, when the model generates its response token by token, the GPU must read the entire historical context’s prefix cache out of GPU memory for every single new token produced. NVIDIA’s aggregate throughput benchmarks on H100 data center GPUs show that stretching active context from 16K to 256K tokens drives the GPU cost per generated output token up by 3.6x to 7x; in single-request tests on DGX Spark desktop workstations, single-token generation latency in the latter half of long runs also roughly doubles on some models. Keeping the window short through periodic Compaction on frontier closed-source models cuts total token consumption substantially while holding code-repair accuracy roughly even with full long context.
If Compaction is such a bargain, why can’t agents in production compact frequently? The engineering reason lies in the destructive cost Compaction imposes on multimodal content and historical caches. According to Anthropic’s on-demand Compaction documentation, once Compaction fires, the server invokes the model to write a plain-text summary replacing older messages, completely wiping out all screenshots, images, PDF documents, and container-uploaded files accumulated up to that point. Worse still, replacing the original message array with newly generated summary text invalidates the message-history prefix cache built up after the system prompt, forcing the next turn to write the cache anew.
In practice, then, a more sensible trajectory for long-horizon agents is a sawtooth wave. Across dozens of consecutive turns focused on solving a single subtask, the agent refrains from Compaction and appends turn by turn via Message Threads and Prompt Caching, preserving historical screenshots and reusing GPU memory caches; once context grows to hundreds of thousands of tokens and the current subtask wraps up, the agent deliberately steps out of the current thread to run Compaction once, folding the long history into a brief summary before opening a fresh thread for the next climb.
The commercial reason hides in API pricing structures. When enterprises buy their own GPUs to deploy open-weight models, expensive long-context decoding burns their own GPU power bills, so NVIDIA speaks from the compute buyer’s perspective in urging teams to compact early and often. When calling Anthropic’s commercial API, however, output tokens are billed at a flat unit price, meaning the API provider absorbs the extra GPU bandwidth cost burned during decoding; conversely, reading hundreds of thousands of historical tokens on every turn of a long conversation incurs a cache-read fee at 0.1x the standard input price, so running dozens of consecutive turns with hundreds of thousands of tokens racked up requires paying for tens of millions of tokens just to re-read old cache. Previously, stalling for several extra seconds every time a user hit Enter in a long chat pushed developers to write code truncating old history; now that Anthropic has cured that lag with gateway memory, developers have less incentive to prune history proactively, preserving hundreds of thousands of billable cache-read tokens on every turn.
Viewed side by side, these recent technologies are each solving pain points at different links along the agent call chain. In late October, the open-source inference engine SGLang pulled session state into the inference serving layer specifically to handle barge-in interruptions at any moment during real-time voice conversations, ensuring cancellation signals actually reach the compute nodes and trim history according to playback progress. For text and tool-calling workloads, Anthropic uses 5-minute expiring gateway memory to save repeated uploads and tokenization; OpenAI switches to persistent WebSocket channels that send only deltas during a task and pair with non-persistent storage to satisfy Zero Data Retention (ZDR) compliance; and NVIDIA uses periodic Compaction to save GPU memory overhead during token-by-token generation.
Once the division of labor across these four approaches is clear, teams building agents can choose according to their own compute cost structure and compliance boundaries. If a team buys GPUs or rents compute to run open-weight models, decode-phase GPU memory overhead is a direct out-of-pocket bottleneck; they should follow the bounded-window approach validated by NVIDIA, setting a short window cap calibrated to their specific model’s compaction tolerance (note that some open-weight models suffer sharp accuracy drops when compressed down to 16K tokens, so the threshold must be selected empirically) and running Compaction regularly.
If a team calls the Claude API directly to build long-horizon
tool-using agents, we recommend immediately replacing per-turn manual
cache_control breakpoints with the global automatic caching
parameter and running a sawtooth pattern: enable Message Threads during
a single subtask’s execution to reduce TTFT on long sessions, then
deliberately exit the thread at subtask transitions to run Compaction
once. For financial and healthcare applications strictly bound by Zero
Data Retention (ZDR) or HIPAA compliance while executing more than 20
tool-calling steps, teams should avoid cross-request in-memory Message
Threads and prioritize evaluating OpenAI’s persistent WebSocket mode
with non-persistent storage enabled, or perform context pruning and
state management locally on the client.