When writing code with AI coding agents today, it is common practice to paste an entire codebase straight into the chat window. Almost everyone has experienced what happens next: sometimes you hit Enter and the cursor blinks for ten-plus seconds before the first word appears; other times, a response that was streaming smoothly suddenly stutters and jerks forward word by word.
The culprit is not a lack of raw GPU compute from your cloud provider, but the fact that the inference engine is running two fundamentally different workloads on the same GPU. To solve this, most production inference clusters now use Prefill-Decode disaggregation (or PD disaggregation). Without adding a single extra GPU, this architecture can speed up live streaming output by more than 3x. Yet a cross-tenant data leak at Cloudflare this fall exposed the hidden engineering risks behind this high-performance design.
To understand why providers split their inference clusters in two, we first need to look at what an LLM actually does when it serves a request. From a user’s perspective, chatting with a model feels like one continuous action. From the GPU’s perspective, however, it consists of two distinct phases: Prefill and Decode.
Prefill is essentially reading the prompt. When you paste a codebase of several hundred thousand tokens into a chat box, the GPU has to compute the attention relationships across all input tokens in one pass, emit the very first output token, and store the intermediate key-value states in GPU memory—what we call the KV cache. Because tens of thousands of tokens participate in matrix-matrix multiplications at once, the GPU’s compute cores run at full tilt while memory bandwidth has plenty of headroom. Prefill is compute-bound, and it dictates how long you wait after hitting Enter before the first token appears.
Once that first token comes out, the model switches to the Decode phase and begins writing one token at a time. On each step, the model computes a single new token and appends it to the sequence to generate the next one. Because only 1 token is processed per step, the heavy matrix-matrix multiplications of the Prefill phase collapse into skinny matrix-vector multiplications.
The catch is that just to generate that 1 token, the GPU still has to read hundreds of gigabytes of model weights and tens of gigabytes of historical KV cache from GPU memory from start to finish. Its compute cores spend most of their time sitting idle, waiting for data to move across memory buses. Decode is memory-bandwidth-bound, and it determines whether streaming output looks smooth to the human eye.
When you colocate both phases on the same machine, they immediately clash. Suppose a GPU is smoothly streaming tokens to dozens of users when a new request arrives with a several-hundred-thousand-token Prefill. That massive matrix computation monopolizes the GPU for several seconds. The dozens of ongoing Decode streams are forced to wait in line, and text on users’ screens freezes like a slideshow. Conversely, if you throttle Prefill to protect ongoing streams, new requests are left staring at a blank screen.
How do you break that deadlock? The intuitive answer is to split the cluster into two dedicated pools: one pool of machines dedicated to Prefill to max out compute, and another pool dedicated to Decode to max out memory bandwidth. In Cloudflare’s inference architecture benchmarks, splitting a fixed number of GPUs into separate Prefill and Decode pools dropped live P90 inter-token latency from a jittery ~100 ms down to 20–30 ms, boosting streaming output speed by more than 3x.
Given that Prefill is compute-bound and Decode is memory-bandwidth-bound, you might wonder whether cloud providers buy two different GPU models for the two pools. In practice, they do not. Maintaining heterogeneous GPU pools in production is operationally painful, so providers typically buy the exact same GPU model—such as the NVIDIA H200—and run different software configurations on each side. Three software trade-offs alone are enough to extract very different performance profiles from identical hardware.
The first trade-off is running different quantization precisions on each pool. Because Decode is bottlenecked by memory movement, the most direct way to speed it up is to shrink the model. In Cloudflare’s model compression and safety benchmarks, compressing model weights from FP8 down to INT4 reduced GPU memory footprint from 700+ GB to around 420 GB, cutting the data moved per token by 40% and making single-stream Decode 55% faster. Quantizing the KV cache down a notch as well doubled the concurrent context capacity per machine and boosted peak total throughput by another 40%.
Yet putting those exact same INT4 weights on Prefill nodes actually hurts performance. Prefill is already starved for compute; with lower-precision weights, the GPU has to spend extra compute cycles dequantizing and unpacking weights before every large matrix multiplication, which reduced measured Prefill throughput by 15%. The sweet spot, therefore, is asymmetric: run larger, higher-precision FP8 weights on Prefill nodes to preserve compute efficiency, and run smaller INT4 weights on Decode nodes to save memory bandwidth.
That asymmetry introduces a subtle side effect. Even though Prefill and Decode nodes use the exact same GPU model with the same total VRAM, the Prefill nodes load much larger model weights—leaving a significantly smaller pool of remaining GPU memory for the KV cache than on the Decode nodes.
The second trade-off is reclaiming GPU memory pages without zeroing out old data. In SGLang, the open-source inference engine Cloudflare uses, a node carves its remaining VRAM at startup into fixed-size cache pages that different users borrow in turn to store their context. A context of several hundred thousand tokens can easily occupy tens of gigabytes of VRAM; zeroing out those pages every time a request finishes and returns them to the pool would waste precious memory bandwidth.
Instead, the engine skips zeroing altogether and relies on an implicit contract for safety: when a new request arrives, the fresh KV cache computed during the Prefill phase will completely overwrite every memory page allocated to it before the Decode node ever reads from those pages. As long as that overwrite-before-read ordering holds, leftover data from the previous user can never leak through.
The third trade-off is letting network cards write directly into GPU memory across machines. Once the cluster is split into two pools, tens of gigabytes of KV cache computed on a Prefill node must travel across the network to a Decode node. To keep the CPU out of the loop, clusters use a transfer engine like Mooncake over one-sided RDMA (Remote Direct Memory Access): the Decode node pre-allocates empty memory pages, and the Prefill node’s network card writes data straight over the wire into the Decode GPU’s VRAM. Each side maintains its own queues and synchronizes with a simple completion signal at the end.
Under normal conditions, these three performance shortcuts work together seamlessly. On an error path, however, they lined up in the worst possible way.
In late October, Cloudflare published a postmortem on a cross-request KV cache leak: for about ten days between late August and early September, the GLM-5.3 coding model hosted on Workers AI suffered a cross-tenant data exposure. After security researcher Gadi Evron at Knostic discovered and reported the issue, Cloudflare took the model offline within 40 minutes of opening the incident and deployed a fix the same day. Their audit found 155 ultra-long-context responses that had been contaminated with fragments of other users’ conversations or source code.
How did the leak happen? In short, the Decode node reserved 8 pages of unzeroed GPU memory waiting to be overwritten, the Prefill node overwrote only the first page before signaling that the entire transfer was complete, and the Decode node proceeded to read the remaining seven pages of another user’s leftover data as if they belonged to the current request. Step by step, the failure walked straight through all three speed trade-offs described above.
The first domino was the mismatched KV cache pool sizes combined with unzeroed memory reclamation. The model advertised a public context window of roughly 1.05M tokens. Decode nodes, running compact INT4 weights, had plenty of free VRAM to fit that window. Prefill nodes, loaded with bulky FP8 weights, only had enough remaining VRAM to hold about 680k tokens (681,280 to be exact). When a user sent an ultra-long request exceeding that threshold—say, one requiring 8 pages of KV cache—the Decode node saw no problem at all and immediately allocated 8 pages from its memory pool to receive the transfer. Because of the second trade-off, those 8 pages had just been returned by a previous user without being zeroed out, and were still packed with that previous user’s source code and chat history.
The second domino was that the Prefill node could not fit the request, yet inadvertently computed 1 token anyway. When the Prefill node received the 8-page request, it realized its KV cache pool was too small and flagged the request as rejected. Normally it would abort and return an error right there, but the inference engine carried over a habit from single-node serving: to avoid disrupting the GPU’s active batching pipeline, it did not immediately evict the oversized request from the queue. Instead, it quietly replaced the hundreds of thousands of input tokens with a 1-token dummy placeholder so the shell request could ride through the current batch and be discarded with an error at the exit.
Moving that single-node trick into a PD disaggregation pipeline proved fatal. The cross-node handoff module missed the rejection flag, failed to recognize that the request was a doomed placeholder, saw that the 1 dummy token had finished computing, and cleared it for transfer as a normal request. That triggered the third trade-off: the Prefill network card fired off a one-sided RDMA write, copied the KV cache of that single dummy token into the first memory page on the Decode node, and immediately sent a transfer-completion signal.
When the Decode node received the completion signal, it neither
checked how many pages had actually been written nor verified whether
the remaining seven pages had been touched. Trusting the contract that
Prefill always overwrites every allocated page, it read all 8 pages of
GPU memory to generate its reply. Page one held the fresh 1-token
placeholder; pages two through eight held the previous user’s leftover
context. The model dutifully answered on top of someone else’s private
code and returned a normal HTTP 200.
One detail of the incident seems counterintuitive at first: SGLang
actually logged the true capacity of the Prefill pool at startup, and
the Cloudflare team had noticed before launch that it was smaller than
the advertised million-token context window. Why did it still ship to
production? Because when the team ran long-context load tests, every
oversized request came back with a clean HTTP 200 and
streaming text. Their load-test script checked only status codes and
latency, not whether the generated answer actually matched the prompt.
Seeing an all-green dashboard, the team assumed the engine was
automatically handling the excess context under the hood via chunked
prefill or similar mechanisms, or that it would throw an explicit error
if it truly ran out of room.
What made the bug even stealthier was that as soon as an oversized request hit the Prefill entry point, it was shrunk down to 1 token—skipping the heavy work of computing 8 pages of Prefill. The failing code path actually responded faster than a legitimate multi-hundred-thousand-token request. And because the Decode node was reading coherent KV cache pages left behind by prior requests, the generated text remained grammatically fluent. On monitoring dashboards, every metric looked green; the more it failed, the faster it ran. Ironically, in early August Cloudflare had demoed a page-ownership tagging prototype with less than 1% overhead that verified page ownership before every read and would have caught this out-of-bounds read cold. After hitting stability issues on older models in mid-August—and because it was still tied to a specific attention backend—the team disabled it shortly before the incident.
Following the incident, Cloudflare updated the model’s Prefill configuration to match its older models, expanding the Prefill KV cache pool to roughly 2.04M tokens (about twice the context window), and worked with the open-source community on a coordinated SGLang security disclosure to patch the engine. The patched version ensures that any request marked as rejected is retired with an error right at the cross-node admission queue before touching memory allocation or RDMA transfer, and Decode nodes now verify upon receipt that the number of transferred tokens matches the reserved capacity.
During the investigation, the two teams also fixed a second, even subtler cross-node race condition inherent to one-sided RDMA in PD disaggregation. A KV cache for several hundred thousand tokens can reach 30 GB and take several seconds to transfer across machines. If a client disconnects or cancels mid-transfer, one-sided RDMA write commands already in flight on the Prefill network card cannot be called back. If the Decode node immediately reclaims those memory pages and hands them to the next user’s request, in-flight packets still traveling across the wire will land moments later and silently overwrite the new user’s GPU memory without leaving a single error log.
To close that window, the new engine enables a deferred KV release mechanism by default: whenever a transfer is canceled or fails mid-flight, both nodes freeze the affected memory pages and wait until the Prefill side drains all in-flight RDMA writes and returns a request-specific acknowledgment before returning the pages to the free pool, backed by new transfer batch limits and end-to-end checksums. In a multi-tenant inference cluster, the happy path is a race for throughput—but the real security boundary is almost always defined by how cleanly you clean up on the abort path.