vLLM and TileRT were not the obvious pair.
vLLM excels at packing multiple requests into the same compute batch, pushing GPU utilization and overall throughput as high as possible. TileRT, built by the Tile-AI team, goes the other way: it is purpose-tuned for batch=1 decode, willing to sacrifice concurrency to shave per-token latency off individual requests. One wants the machine to do more work; the other wants this request, right now, to go faster.
On 14 July 2026, the vLLM project published a blog post announcing disaggregated inference integration with TileRT, shipping alongside TileRT 0.1.5. The approach described in the official technical blog is a division of labour: the two engines do not merge codebases. Instead they connect into the same serving system through a public connector. vLLM reads the prompt, completes prefill, generates the first token, then hands subsequent decode work over to a TileRT execution pool.
What I find most interesting about this integration is not that vLLM gained another backend. It is that two engines with opposite goals are now running behind the same service. Until recently, the natural habit was to compare engines and ask which one scores highest on aggregate. But real-time interaction, long-context workloads, and speech generation are pulling in different directions, and a single runtime is finding it harder to handle every request well. The competition in inference serving is shifting from “one engine, best overall” to “specialised execution paths — and the ability to orchestrate them.”
Why can’t one engine do all of it? The answer lies in how a model produces a response.
Before the model emits its first token, the GPU must read the entire input and build the KV cache needed for subsequent generation. This step is called prefill; it is usually compute-bound, and it directly determines time-to-first-token (TTFT). Once that first token is out, the work rhythm changes. The model enters decode, generating one token at a time, and every step must read model weights plus the existing KV cache. Decode has lower compute intensity than prefill; memory bandwidth tends to become the limiting factor earlier, and users experience the result as per-token output speed, commonly measured as TPOT.
That is where the trouble starts. High-concurrency services usually batch many requests together so that overall throughput goes up, but individual requests may have to wait and can be affected by what else is in the batch. Switch to batch=1 and you reduce queueing delays and cross-request interference, yielding lower TPOT — at the cost of typically lower overall throughput and hardware utilisation.
The consequence: the same hardware and the same scheduling policy cannot easily deliver both high batch utilisation and low queueing with low interference. When text workloads are relatively homogeneous, a general-purpose engine can find a workable middle ground. Once real-time interaction, long-context processing, and speech generation enter the same service, that middle ground stops being enough.
The answer vLLM and TileRT settled on is: if the trade-off is painful, keep both specialised solutions and use a control plane to split and route.
vLLM stays at the service entry point, handling prefill, prefix caching, and scheduling. Only at decode does the traffic fork: conventional high-concurrency requests remain in the native vLLM instance pool, while requests flagged for low latency are handed to TileRT through vLLM V1’s public connector, MultiConnector. Both decode pools share the same prefill service. The entire setup does not require forking or patching vLLM.
There is a detail here that is easy to miss: a request must be explicitly flagged before it ever reaches TileRT. The current integration solves the problem of how two engines share prefill and hand off generation state, but it does not decide, on behalf of the platform, which requests should use the low-latency pool. Unflagged traffic stays in the native pool. Service providers still need to devise their own routing policy based on product tier, request shape, queue depth, and TileRT capacity. The connector has opened the road; deciding which vehicle belongs on which lane is still the job of the system above.
This division means TileRT only has to handle the workload it is built for. In our earlier article on TileRT, we examined how it arranges fine-grained operators, I/O reads, and inter-card communication into a continuous execution stream for batch=1 scenarios, minimising gaps between kernels. The integration currently supports models including GLM-5/5.1 and DeepSeek-V3.2.
Speed has another side: capacity. In TileRT 0.1.5, a single 8×B200 decode node processes exactly one in-flight request at a time. This constraint explains how it can focus so intently on single-request latency, and it is also a reminder not to apply conventional concurrency arithmetic to this path.
The interface looks unified, but functional coverage is not yet
complete. According to the open-source
code, the routing script pd_router.py passes only three
sampling parameters — temperature, top_p, and
top_k — to TileRT after the first token. Other sampling
capabilities have not yet been proven semantically consistent across
this hand-off path.
RDMA transfers that ship data between nodes run asynchronously in the background, but extracting state data such as the KV cache on the source node is still synchronous. In a production network, the overhead of moving state could eat into some of the operator-level gains. The data shared with this release shows generation speed on 8×B200, but there is not yet an independent end-to-end test of the connector itself. The vLLM project’s disaggregated prefill documentation also makes it clear that disaggregated deployments do not automatically improve overall throughput — their primary use is to provision resources for the two stages separately and tune TTFT, TPOT, and tail latency independently.
So the existing data does tell us that the TileRT decode path itself is fast. It does not yet answer how many requests a production system can serve per dollar. To do that arithmetic, prefill, state extraction, network transfer, queueing, and idle nodes all have to be accounted for. Low latency might also become a sound economic choice only when the high-value requests are enough to fill the dedicated capacity.
vLLM x TileRT does not eliminate the latency-versus-throughput tension. It preserves two independently optimised paths and moves the choice into the control plane — into routing, capacity planning, and cost accounting.
If specialised paths only meant separating different users, the story would not be that complicated. Real-time speech models go further: a single request already contains several computations that run at fundamentally different rhythms.
A speech reply typically passes through several stages: the Thinker generates text tokens at a low frequency; the Talker generates codec tokens at a high frequency; and finally a convolutional Vocoder turns the code back into an audio waveform. Every step the Talker takes must also tightly feed back into a multi-token prediction module (MTP). They are chained within a single reply, but their compute characteristics are not the same.
The Talker-MTP loop is especially troublesome. Each step is lightweight, and the time spent on kernel launches and inter-card synchronisation can easily exceed the time spent on the actual operators. At that point, further optimising an individual matrix multiplication yields diminishing returns; what really needs to shrink is the dead air in the execution stream.
If all these tasks are packed into a single scheduler, when one stage gets busy the others are forced to wait. The SGLang Omni project therefore gives the Thinker and Talker independent schedulers, while keeping the tightly coupled Talker and MTP inside the same forward pass to avoid cross-stage communication. An outermost Coordinator manages only the pipeline topology.
Independent schedulers let each stage batch and parallelise at its own tempo, without waiting for the entire speech pipeline to synchronise. Data moves between stages via a unified inbox, outbox, and zero-copy shared memory. This preserves the overall topology of a multi-stage model without forcing every stage to accept the same execution policy.
This points in the same direction as TileRT. As the SGLang Omni analysis noted, when each computation step is lightweight, what the system needs most is to close the gaps between kernel launches and synchronisation. Specialised execution paths are now moving from between requests to inside a single multimodal model.
TileRT specialises by latency target. TensorRT-LLM demonstrates a different choice: if the model and the hardware are already fixed, accept a heavier build cost and go deeper on optimisation.
When the model architecture is stable and the deployment environment is certain to use NVIDIA GPUs, this trade can make sense. The team pays model build, compilation, and tuning costs in return for optimisations that target a fixed platform. In TensorRT-LLM’s disaggregated serving design, context (prefill) and generation (decode) can be assigned to separate GPU resource pools and can use different parallelisation strategies.
Splitting the pools reduces resource contention between the two stages and allows teams to tune TTFT and TPOT independently. That said, the official documentation also notes that the overhead of transferring KV cache across nodes cannot be ignored, and the benefit depends on input-output shape. Long-input, medium-output tasks may see a clear gain; short-context interactions may lose more than they win. Each context or generation instance can expose an OpenAI-compatible server, with an outer disaggregated server coordinating through a REST interface.
The cost structure on this path differs from TileRT’s. TileRT concentrates its optimisations on batch=1 decode; TensorRT-LLM lets teams trade higher build cost and deeper platform lock-in for thorough optimisation of a stable model on NVIDIA hardware. Both are giving up generality — they just draw the boundary in different places.
Once there are multiple paths, the trouble multiplies: how does the platform pick the right one for each request?
An OpenAI-compatible API can unify the client-facing endpoint, but it does not guarantee that backends behave identically. The NVIDIA Dynamo feature matrix shows that different backends support request cancellation, multimodal input, dynamic LoRA loading, speculative decoding, and KV cache management to different degrees.
When a request reaches the gateway, the router must first check requirements such as sampling parameters, tool calls, and mid-stream cancellation. If the target backend does not support even one of those, the request cannot be sent there. A unified API answers “how to call”; capability validation answers “if we switch backends, does the meaning change?”
After capability matching comes state continuity. A request that completes prefill in vLLM and then switches to TileRT or TensorRT-LLM for decode must deliver the KV cache, sparse indices, and draft state that the target node needs to continue generating. Transport layers such as NIXL and Mooncake can handle the movement, but whether the state extraction is complete, and how long the transfer takes, both affect the outcome.
The inference gateway of the future cannot just load-balance by connection count. It needs to consider request shape, backend queue depth, KV cache affinity, and the actual capabilities each backend exposes, then estimate which path can finish the request on time. When a specialised path fails, it also needs to fall back safely to a general-purpose path.
When something goes wrong, the team must know where to look. The platform cannot merely log which node handled the request; it needs to record routing rationale, state-transfer latency, capability degradation, and fallback outcomes. Otherwise, even if average performance improves, it will remain difficult to say whether a single slow request was stuck on queueing, transfer, execution, or a bad routing decision.
In the past, when people talked about inference engines, the most natural question was “which one runs fastest on the benchmark?” Going forward, the question that will come up more often is likely to be “which execution path should this particular request take right now?”
Whether a specialised low-latency runtime like TileRT will persist as a separate project in the long term is impossible to say today. vLLM, SGLang, or TensorRT-LLM may gradually absorb these optimisations and come to cover more execution paths themselves. Even if the implementation boundaries shift, the underlying premise — that workloads have already diverged — will not go away. Whoever can cover the most specialised paths, and let requests switch between them correctly and at low cost, will be closest to the centre of the next-generation inference platform.