The machine under my desk is packed with 3 GPUs.
Running on just a single card, single-GPU decode throughput (the phase where response tokens are generated) exceeds 200 TPS.
Yet when multiple cards run together, things change. I had an agent spawn several concurrent branches to read files, look up information, and invoke tools. In theory, 3 cards should divide the work cleanly. Instead, the terminal output kept stuttering, with all branches bottlenecked together. Switching to the monitoring dashboard, none of the 3 cards were maxed out: one card was smoking hot with work, while the other two sat idling nearby.
The slowdown wasn’t because any card lacked compute, but because these branches were all assigned to the exact same card. When running multiple cards together, something has to decide which card gets each task—and this “how to divide” is the crux.
This machine houses 3 5090 cards with 32GB VRAM each, with no NVLink between them, connecting solely over motherboard PCIe. Inter-card communication is slow, so instead of splitting the model across cards, I loaded a full copy of the weights onto each card, running a resident 4-bit quantized Qwen3.8-27B instance. The backend is the local engine NInfer, paired with MTP speculative decoding; with thinking mode turned off for chat and coding, single-stream decode runs at 150 to 256 TPS.
In the community, comparing GPU speed usually means feeding a batch of mutually independent, fixed-length prompts, clearing each upon completion to measure tokens generated per second. This standardized speed test is benchmarking. On this machine, feeding 1 to 8 concurrent independent requests, the 3 cards combined reach 268 to 962 TPS—impressive numbers on paper. But real-world agent workloads are another story: a single task requires chatting across many consecutive turns, each triggered by the previous one, snowballing with the entire history as it grows longer and longer; tool calls introduce pauses in between; and the main agent can fork into multiple parallel branches at any time, all sharing the exact same system prompt and background at the start. That is precisely why several recent benchmarks tailored specifically for agentic traffic make “cross-turn history reuse” a mandatory requirement (example); fixed-length independent benchmarks are no longer enough.
Long conversations throw multi-GPU setups into disarray, rooted in the GPU’s two-step pipeline for processing long texts. The model cannot remember what was said in the previous turn. With every new follow-up question, the program must concatenate the entire history from all preceding turns with the new message into a long prompt spanning tens of thousands of tokens, feeding it to the GPU anew. The GPU operates in two steps: first, it reads through the entire input from scratch to compute contextual representations across all layers—this is called prefill, which consumes compute; second, guided by these computed states, it generates tokens one by one—this is called decode, which consumes memory bandwidth. By turn 20, the preceding 19 turns haven’t changed by a single character. To avoid re-reading every turn, the GPU retains intermediate states from each layer in VRAM during prefill for later lookup; this cache is the KV cache. When the next turn arrives, as long as that long opening history exactly matches what is stored in VRAM, the GPU directly retrieves the KV cache to skip re-reading, processing only the few newly appended sentences at the end—this is called prefix reuse. A public benchmark demonstrated: on a single instance with a 10,000-token prompt, the first run without cache required prefill from scratch, resulting in a time to first token (TTFT, the latency before the first token appears) of 4.3 seconds; the second run hit the KV cache and completed prefix reuse, dropping TTFT to 0.6 seconds. Some agent teams go so far as to call cache hit rate the single most critical metric for production-grade agents.
With only one card, requests naturally land on the same card, getting reuse as long as VRAM permits. Once scaled to 3 independent cards, a front-door scheduler must decide which card gets each new task. A typical website backend is stateless—round-robin dispatching or picking the least busy machine suffices. But in LLM serving, each card accumulates the KV cache of different conversations. If you casually dispatch a long conversation to an idle card, that card lacks the preceding cache and must recompute the prefill of tens of thousands of tokens from scratch. This dispatching mechanism—which must both check which card is free and identify which card holds the existing cache—is routing.
What makes routing difficult is that two objectives fight each other: to reuse existing cache, requests with identical prefixes must consistently go to the card holding that cache; to avoid queuing, incoming requests arriving simultaneously must be distributed evenly across the 3 cards. The former is called cache affinity, while the latter is called load balancing—one pushes requests toward the same card, while the other disperses them across multiple cards. When several branches launch concurrently, they often share the exact same system prompt at the beginning. Once a card caches this common prefix, a scheduler biased toward cache affinity dumps all subsequent branches onto it, leaving one card queued up while the others sit idle. The industry refers to this clustering as request herding (analysis). Research has also proven that a single mapping rule cannot satisfy both cache affinity and load balancing simultaneously; blindly pursuing higher hit rates will inevitably overload instances hosting popular prefixes. The proposed remedy is to calculate two candidate card slots for each request simultaneously, then select the better fit based on current load and latency targets—effectively striking a compromise between the two goals.
Why those initial branches all piled onto a single card became even clearer on my own machine. The front door runs the official sglang-router with its default cache-aware policy enabled. These branches shared the same system prompt at the beginning. After the first card cached this common prefix, the router locked onto it, funneling subsequent branches right back there and ignoring the other two idle cards. It behaves this way because it can only inspect request text, defaulting to prioritizing cache reuse for requests with matching prefixes; it only switches cards when the queue difference between two cards exceeds a certain threshold. If the gap isn’t wide enough, it stays stuck.
Another scenario is even more troublesome: when a long conversation approaches the context window limit, the client compresses and rewrites earlier history, swapping out a few sentences in the middle. NInfer determines “whether this prefix can be reused” based on the history being completely unchanged; the moment something in the middle changes, it has no choice but to treat the entire history as new and re-run prefill from scratch, instantly causing stutter.
Moreover, agents work in bursts: one second the cluster is completely idle, and the next second the main agent might spin up multiple concurrent branches, saturating all 3 cards at once. To prevent these sudden spikes from creating queues, I typically have to leave more than half of the cluster’s compute capacity in reserve, preferring idle GPUs in exchange for smoothness.
Why would a front-door router leave two idle cards untouched, insisting on cramming two long conversations onto a single card to fight for resources? Because it dispatches tasks from outside the room, unable to see what is actually stored in GPU VRAM or how much work is currently backlogged, left to blindly guess through the keyhole. Traditional load balancing assumes backends are stateless, requests are independent, and nodes are interchangeable—all three of which are completely broken by LLM inference (survey). To route tasks accurately in data centers, the scheduler must track two categories of information from every card at all times. The first is the actual cache inventory in VRAM: whenever a GPU allocates or evicts a block of KV cache, the engine fires an event, allowing the scheduler to maintain a cluster-wide cache map tracking exactly which text resides on which card. The second is the backlogged workload on each card: merely knowing how many re-reads can be avoided isn’t enough; it must also account for how many input tokens are still awaiting processing and how many in-flight decode steps are queued up. This measure of busyness, quantified in tokens waiting to be processed, is called token load. Combining both sets of information, the scheduler converts un-cached tokens plus existing token load into an estimated waiting time, selecting whichever card will finish fastest.
With the engine reporting real-time cache mutations and token load, multi-GPU routing in data centers has matured considerably. The typical approach involves engines proactively emitting events while the scheduler picks cards based on composite cost, which in benchmarks has accelerated time to first token by 57x and doubled throughput (architecture).
Yet this entire stack remains out of reach for individual builders.
It demands vLLM or SGLang under the hood, deployed on Kubernetes, with
engines reporting KV cache events in real time. Individual multi-GPU
users, aiming to squeeze out maximum single-card speed, often opt for
specialized engines: the NInfer engine I use exposes only a
/health endpoint, offering no way to query cache state or
in-flight load; another common open-source engine (example) handles multi-GPU
by splitting layers across cards within a single machine, with cache
managed locally by individual nodes. This solves the mystery of why
those branches were sucked onto the same card: the front-door scheduler
can only watch incoming request text from the outside, assembling an
approximate prefix tree on its own. Unable to see the real cache in VRAM
or accurately assess which card is truly optimal at that moment, it
blindly sticks to the card that seems most likely to hold the cache. The
community has seen routers designed for self-hosted clusters that
attempt to maintain a prefix index at the gateway layer using their own
tokenizer, multiplying hit rates several-fold and slashing median TTFT
by nearly 30% (project); but
as long as the underlying engine remains a black box, guessing internal
state from outside will always have a hard ceiling.
Taking this thought one step beyond routing: even if multi-GPU dispatching were tuned to perfection, could the unit cost of this rig under my desk actually undercut cloud APIs billed per token? Many people build GPU rigs based on the intuition that hardware is a one-time purchase, surely cheaper than paying ongoing API fees. But once hardware depreciation and electricity costs are amortized over generated tokens to compute the cost per million tokens, and then compared against cloud bills, that intuition falls apart fast.
First, look at how aggressively commercial APIs have driven down prices: DeepSeek V4.1-Flash is a 552B mixture-of-experts model activating 8B (input) and 16B (output). According to official pricing, off-peak output costs 0.60 dollars per million tokens; when input hits cloud KV cache and completes prefix reuse, it receives an automatic 98% discount, dropping as low as 0.003 dollars. When redundant computation is eliminated to this extent, prices naturally plummet.
Now let’s calculate my three-card rig. The 5090 is nowhere near its launch price anymore: in October 2026, a brand-new card runs around 5000 dollars in the US, and used ones hover around 3800 (price tracker), putting 3 cards plus chassis and power supply at roughly 17000 dollars. Amortized over 4 years, fixed monthly costs are about 350 dollars, with full-load power consumption around 1.7kW. Only under 24-hour full utilization, factoring in 3 cards combining for 600 TPS, does the cost per million tokens drop to around 0.4 dollars; amortized across an individual’s realistic usage of 1 to 2 hours per day, it immediately spikes to 4 to 6 dollars—nearly 10 times more expensive than calling an API.
On the exact same machine, why is the theoretical unit cost 0.4 dollars while daily reality is four to six dollars? The key isn’t how fast the GPUs compute for those few seconds, but how many hours out of 24 hours a day the machine is actually working—this ratio is utilization. Buying hardware or renting cloud instances charges by time; calling APIs charges by tokens actually generated. A cost analysis of self-hosting shows that an always-on H100 on AWS costs roughly 5000 dollars a month fixed, requiring 9% utilization just to break even with APIs; another report points out that average daily utilization for many clusters drops as low as 5% (data). On an individual multi-GPU rig, two more discounts compound: running only 1 to 2 hours a day leaves the machine idle most of the time—that is the first layer; even while working in front of the computer, I have to intentionally keep over half the compute capacity idle to prevent routing collisions—that is the second layer. Stacking both layers together, the headroom preserved earlier for a smooth experience ultimately ends up on the balance sheet as pure overhead.
Laying out both the engineering hurdles and the economics makes the true positioning of self-hosted LLMs clear: shoving a multi-GPU machine under your desk is like keeping a diesel generator at home. Nobody buys a generator to save on electricity bills; once depreciation and fuel costs are amortized over every kilowatt-hour, home-generated power can never compete with the grid. The reason we gladly tinker with multiple GPUs, tune inference engines, and obsess over routing strategies is to buy three things that APIs simply cannot offer: keeping code and data strictly local, running uninterrupted even when offline or rate-limited, and the hands-on control and joy of pulling the system apart to see exactly how caching and scheduling mesh together.