tok/s Is the Most Deceptive Performance Metric in Agentic Scenarios

In my daily agentic coding workflows, I routinely switch back and forth among a dozen models and multiple providers. Getting Qwen 3.8 27B running locally this year brought a tangible shift in experience—it is the first small model I have used that truly crosses the intelligence threshold. Running on my local workstation, it responds briskly, rarely hits a wall, incurs marginal costs that are practically just electricity, and a session can run steadily for hours.

Once everything was running smoothly, a technical question came up. If we take an already smart-enough small model and push its inference speed to the physical upper limit commercially available in the cloud today, how much better does the experience actually get? Is there a qualitative leap? To test this idea, I hooked into the same 27B weights hosted on Cerebras (as for why their chip is so fast, I wrote a dedicated piece earlier). The vendor claims it can reach around 1,500 tokens per second. Compared to the 102 tokens per second measured locally, on paper there is a generational speed gap of nearly 15×. Following my usual workflow, I plugged it directly into my real-world coding pipeline.

In actual testing, it was indeed more than 3× faster. But that speed boost brought other problems: a smaller context window, far more frequent rate limits, and a price tag dozens of times higher. The entire session lasted only three minutes before grinding to a halt, racking up a $1.57 bill in the process.

Fast, but Not as Fast as You Think

Let’s start with speed itself. The vendor’s advertised 1,500 tokens per second looks impressive on marketing pages. But sitting in front of the terminal waiting with the naked eye for each round of output, I didn’t feel a 15× leap—in practice, it was only a little over 3×.

The reason for such a discrepancy is that the entire industry reports tok/s using short context. Standard industry benchmarks mostly run on short contexts between 1K and 4K, measuring only token generation speed during the pure decode stage. There is an easily overlooked detail here: whether it is prefill or decode, throughput is a function of context length—the longer the input prompt, the slower it runs. In real multi-turn development tasks, what I am actually waiting for is the entire end-to-end latency, including input prefill.

In my own agentic workloads, the P50 context across models falls between 50K and 120K, with some long-context sessions surging to 200K or even higher. The input context in a single turn is often 100 to 600 times the output tokens of that turn. For example, my measurements showed 412× for gpt-5.6-sol, 631× for ollama-cloud/glm-5.2, and 227× for locally run Qwen 27B. This is the distribution measured in my own workflow and does not represent an industry average. When the ratio is this skewed, prefill time becomes a substantial portion of the end-to-end wait—for Cerebras, it even accounted for 55% of the median duration.

The metric I track is actual usable throughput—the real speed at which you wait with the naked eye for each round of output. Prefill is counted in, tool execution time is stripped out, and we take the median P50. Below is the primary baseline data recorded on my workstation over 30 consecutive days, where n is the sample count (filtered generation count, not session count):

Dividing the two medians, 357.4 by 101.7 yields an actual ratio of about 3.5×. Cerebras is indeed faster, but falls far short of the paper-spec generational gap of nearly 15×. Breaking down prefill and decode reveals where the speed difference lies, as shown in the table below:

Looking strictly at pure decode, the gap between the two is indeed 6.6×, and the vendor’s claimed 1,500 tokens per second holds up in order of magnitude (in a short-context subset under 20K, it even reached 1,854). But once back in real-world long-context workflows, long-context prefill consumed most of the cloud solution’s decode dividend, narrowing the actual speed ratio to 3.5×. And for the faster model, prefill ate away an even greater share of the advantage: Cerebras dropped from 1,040 to 357, while local only fell from 157 to 102.

The vendor’s claimed order of magnitude matches measurements under short context, but that isolated metric tests only short-text decode and does not directly translate into the real experience of multi-turn interaction. Discussing generation speed in isolation from specific context ratios easily skews model selection.

The Faster It Is, the Faster the Session Dies

The 3.5× speedup is real, and it is definitely noticeable in daily work. But speed itself can cause problems: what truly broke the experience was the chain reaction when speed collided with the agentic loop.

The agentic loop works like this: each turn, the model re-reads the entire bundle of previous file contents, tool outputs, and conversation history. Over multiple turns, the input context snowballs. The faster the model generates, the more frequently multi-turn iterations advance, and the greater the volume of tokens sent to the server every minute.

Looking through the execution logs of those three minutes reveals the entire breakdown process. After the first session launched, context length surged from 11K to 99K in just 65 seconds. The model finished each round of output in one or two seconds, after which the client immediately triggered tool calls and moved to the next turn, causing the tokens sent per minute to skyrocket. On the Cerebras Developer tier, qwen-3.8-27b has limits of 150K un-cached TPM, 450K total TPM, and 450 RPM. In roughly 65 seconds, the first session sent 13 consecutive requests, with ~102K un-cached input (68% of the 150K limit) and ~3K output, but the history carried across multiple turns drove cache.read to 760K (corresponding to an 88% cache hit rate for that single session). Total cumulative tokens sent to the API reached approximately 865K—192% of the 450K total limit—directly triggering a 429 error "Tokens per minute limit exceeded". What actually killed the request was the total TPM bucket; the un-cached limit (102K / 150K) still had plenty of headroom.

Here lies a counter-intuitive mechanism design: cache reads are cheap on the bill, but in rate limits they still count in full toward the total TPM budget. The agentic loop must resend the continuously expanding full history every turn; a high cache hit rate cannot curb the near-linear growth of total token volume, and those 760K of cache re-reads were precisely the main driver pushing the per-minute total to 865K.

Subjectively, it felt like hitting a wall every 15 to 30 seconds. That subjective impression aligns tightly with the hard metric of a breakdown in 65 seconds in the backend.

Now consider the actual bill incurred during those 3.24 minutes. In that brief span, the calls generated 34 assistant messages, bringing the total bill to $1.57. Billed input reached 1,534,667 tokens, costing $1.52 at $0.99 per million tokens; billed output reached 34,935 tokens—comprising 6,866 base output and 28,069 chain-of-thought reasoning—costing $0.05 at $1.49 per million. Among billed input, 82% came from cache reads (specifically 1,264,128 / 1,534,667). Hourly, this translates to nearly $29 per hour, whereas running locally at full load costs about 9 cents per hour in electricity. There is a common thread here: both cost and TPM are input-dominated; what dictates both ledgers is what gets sent in. Even with an 88% cache hit rate, the un-cached input portion (270K) was still 7.7 times my actual output (35K). Caching can save on re-reading historical context, but it cannot save on genuinely new content entering each turn. Laying this out against electricity costs is even more revealing: for that same 3.24 minutes of workload, running locally takes 11 minutes and costs roughly 1.7 cents in electricity (full system load of 500W, residential rate of $0.1844/kWh); Cerebras charged me $1.57—nearly 90 times as much.

Pushing speed to the extreme bought a faster collision with per-minute budget ceilings and a hefty bill. What those $1.57 yielded was an aborted session, leaving the coding task at hand ultimately unfinished.

Why “Good Enough” Wins

So what actually counts as winning? The ultrafast cloud solution had the edge in raw point-speed, but lost in overall performance. This demonstrates that evaluating whether a model is usable in an agentic setup cannot rely on a single speed metric alone. I summarize practical usability into an overall evaluation formula: usable = (intelligence ≥ threshold) × usable throughput × context lifespan × cost × reliability

In this formula, intelligence is a threshold: if it fails to clear the bar, whatever numbers follow are meaningless; once it clears the bar, smarter is better, but it functions as a switch rather than a linearly stackable multiplier. A model below the passing line throws frequent tool call errors and repeatedly derails on instruction execution; no matter how high its numbers are along other dimensions, it cannot be put to practical use. Once it clears the bar, the model can shoulder daily development tasks. Qwen 27B smoothly clears this threshold, and the ultrafast cloud service satisfies it as well; the watershed between the two comes down primarily to the remaining four metrics.

Comparing the two approaches across dimensions side by side:

Comparing the four dimensions, local deployment holds the advantage across cost, context lifespan, and reliability, trailing only in usable throughput. The ultrafast cloud service won on throughput numbers, but fell behind on the other three. Winning on speed alone cannot make up for losing on the other three dimensions. Once you factor in the real costs of agentic workflows, the rankings flip.

Local deployment provides a true 256K context window, costs around 9 cents per hour in electricity at full load, and sessions run steadily for hours. This kind of steady, continuous execution capacity is far more practically valuable than bursts of tokens over a few seconds.

To be fair, there is one category of work it cannot handle. For difficult tasks that span hours and rely on massive context to maintain complex planning and deep reasoning, top-tier frontier models still retain an irreplaceable reasoning edge. Qwen 27B handles mainstream daily development tasks competently, but it is not the ultimate answer for long-chain, highly complex heavy lifting.

Where These Numbers Come From + Don’t Buy tok/s

All the judgments above come from measurements on my own machine. The measured data points throughout this article are drawn from real operational logs. Usable throughput is calculated as output tokens plus reasoning tokens divided by generation time. Timing starts when a single-turn message is initiated and runs until generation finishes; if that turn ends by initiating a tool call, elapsed time is truncated to the moment the earliest tool begins execution. The runtime of the tool itself is excluded from model duration, while time-to-first-token (TTFT) is fully included. The filter criteria require at least 50 tokens per output and a duration between 0.2 seconds and 1 hour, ultimately reporting the P10, P50, and P90 percentiles.

This dataset spans all daily real-world calls across 17 models over 30 consecutive days, comprising roughly 73,000 authentic samples from my daily coding and task execution, with zero synthetic benchmark runs. Ranking the 17 models by P50 usable throughput, the locally deployed 27B model happens to be my most responsive daily workhorse in high-frequency interactions. The specific comparison is as follows:

gemini-3.6-flash ranked fastest among cloud models in measured speed; locally deployed qwen3.8-27b followed closely behind with over 11,000 samples; gpt-5.6-sol bore the heaviest workload (24,211 samples), with a lower median usable throughput due to extensive chain-of-thought processes in its output. This distribution demonstrates that practical rankings based on real usable throughput diverge dramatically from vendor theoretical speed leaderboards.

For the same GLM-5.3-Flash weights, Ollama Cloud measured 70.3 tok/s while Z.ai reached only 19.7 tok/s—a 3.6× gap. Under identical weights, the serving stack determines how much speed you actually get, which is why I use Ollama Cloud more often in daily work. An unexpected finding is that Z.ai’s GLM-5.3-Flash (19.7) was slower than its own GLM-5.3 (40.1), likely because its domestic silicon stack is still being tuned. However, with n=53 for Z.ai Flash being relatively small, this comparison is directional rather than definitive.

My advice for peers evaluating models is straightforward: when selecting models for agentic scenarios, don’t fixate on advertised token generation rates. A more practical approach is to measure end-to-end usable throughput at context depths matching your actual workloads, weighing the true capacity of the context window against invocation costs.

The companion query tool has been open-sourced in the opencode_skill repository. Built as a read-only command-line utility, it directly inspects local SQLite databases. A single command reproduces full reports, exports Markdown analyses, and generates charts. Running it on your own authentic historical workloads will often surface plenty of unexpected findings.