The premise of Model Routing is straightforward: to organize models of differing capabilities, speeds, and costs within a single AI system, with a Router dynamically selecting the executor based on the request. Strong models and small models typically differ by 10 to 50 times in token cost and by several times in latency. If the Router can judge accurately — routing simple mechanical work to a low-cost small model while reserving complex decisions for the strong model — the system can maintain output quality while reducing both cost and latency. The prospect of achieving comparable overall experience at lower cost has attracted extensive research and engineering efforts.
Academic and startup teams have made many attempts in this direction. Stanford’s FrugalGPT and UC Berkeley / LMSYS’s RouteLLM demonstrated cost-quality trade-offs on standard single-turn Q&A benchmarks; Martian (which raised a $9M seed round), Not Diamond ($2.3M seed round), and Unify (YC-backed with an $8M raise) have attempted to package dynamic model selection as a standalone service. On the open-source tooling side, gateway routers such as OpenRouter Auto Router and LiteLLM have also become common entry-layer components.
But this logic, which works smoothly for single-turn Q&A, encounters significant real-world resistance once it enters Agent-style continuous conversations and tool-calling scenarios. This article explores: why does this seemingly straightforward idea consistently stumble in real Agent workflows? Where exactly do the intelligence challenges and engineering bottlenecks lie?
To understand the root cause, we should first review how early Model Routing worked.
Previous mainstream approaches mostly made routing decisions at the level of a single-turn query. The Router received the current prompt, assessed its semantic difficulty or classified its intent, and directly output a Model ID. This mechanism works smoothly for mutually independent Q&A, translation, or information extraction tasks — each request is isolated, and there are no constraints between which model handled the previous turn and which model handles the next.
But when the task involves continuously modifying files, calling tools, and progressing through multiple conversation turns toward the same objective, switching models is no longer just picking a single executor — the accumulated context, tool state, and model-specific caches all change along with that choice.
vLLM
Semantic Router issue #1439 documents a concrete failure case: a
user was performing a complex Go code refactoring task. The strong model
had advanced smoothly through the first several turns. Then in the
fourth turn, the user typed just four English words:
looks good, commit it.
The Router, relying on the semantic difficulty of these few words
alone, judged the input to be both short and not requiring complex
reasoning, and routed it to qwen2.5:0.5b. The small model
replied with pleasantries without advancing the commit task. The issue
was not the semantic difficulty of those four words per se, but that
they must carry forward the task state established in the preceding
turn. Even if the Router’s difficulty assessment for
commit it were corrected, as long as the Router only looks
at the latest message, similar failures will recur.
The commit it failure shows that when routing moves from
single-turn Q&A into multi-turn conversations, the system first
faces a reframing of its cognitive perspective — if the Router cannot
see the full task progress, the algorithm cannot make a sound decision.
When commit it arrives at the routing layer, how much
context the system can observe, what signals it relies on for judgment,
and what actions it can execute directly shape the final routing
decision.
The first constraint on routing is observation scope. When only a single query is visible, the system can only access the current input and has no visibility into the full task progress. Without knowing which model was previously used or how far tool invocation has progressed, routing decisions become divorced from real context. Extending the observation scope to multi-turn history increases not only the input token count, but more importantly allows the Router to connect the current request with the prior task state.
The second dimension is the mismatch in judgment basis. Per-request routers are trained to assess the semantic difficulty or textual similarity of the moment. For stateful routing, the decision must also account for conversational evidence: which model handled the previous turn, whether the current task is in a tool-execution phase, and whether switching models right now pays off.
The third dimension is the expansion of output actions. Early routers mainly output a Model ID or decide whether to escalate to a stronger model. Later approaches extended this to choosing processing paths or orchestrating collaborative execution. But as long as the observation scope remains confined to the current query, no matter how rich the action vocabulary becomes, it still cannot answer whether it is safe to switch right now. Only by incorporating session state into the control loop can routing output expand from a single target model to decisions like keeping the current model, locking the current phase, or spawning a sub-task.
After understanding the intelligence-level reframing of the routing perspective, the next layer to contend with is the underlying physical bottleneck. Even if the Router has the cognitive scope to see the full context, actually executing model switches in a live session is still constrained by hard engineering and hardware limitations. Switching models mid-session is far more than just passing context along — every time a model switch is triggered, the system must face these four concrete engineering constraints:
Historical format incompatibility and out-of-distribution risk are the first obstacle encountered when switching. The moment the Model ID changes, the associated prompt format and tool-calling syntax may also change. For the new model, the historical transcript generated by another model is not only out-of-distribution — it may even contain tool formats or function-calling syntax that the new model cannot parse, causing subsequent generation to break down completely.
Cache invalidation and the resulting re-reading overhead are an
invisible economic loss. In long-context tasks, Prompt Cache is key to
reducing both latency and cost. Service providers like Anthropic
explicitly list model_changed as a cause of cache miss.
After switching models, the target model must re-load and re-parse the
lengthy prefix history, causing the first turn after the switch to incur
a steep overhead spike. The marginal savings from frequent mid-stream
switches are easily wiped out entirely by the re-reading cost of cache
misses.
Invisible implicit state that cannot be transferred further aggravates the discontinuity. The visible message transcript does not necessarily contain all runtime state. Claude’s Extended Thinking signatures, OpenAI-compatible reasoning items, and Gemini’s thought signatures all carry model- or provider-specific continuation information. Official documentation only guarantees compatibility along specific paths — it does not promise that these implicit reasoning chains can be freely moved across models or providers.
The coordination overhead of multimodal and tool-generated artifacts is another complex burden that cannot be ignored. Multiple models decide who executes what, while multimodal content describes the code, screenshots, audio, and other tool artifacts flowing through the task. When different modalities are sent to specialized models, verifying and seamlessly merging the structured results produced by different models back into the main task often incurs extra engineering glue costs that far exceed those of single-model execution.
These engineering constraints compound each other: switching to a more specialized model may improve the current step but introduces cache cold-start and state migration penalties; compressing context to ensure compatibility with the history may in turn weaken the information needed for the new model to make sound judgments.
The upper-layer reframing of cognitive scope and the lower-layer engineering constraints are intertwined, making the ideal calculus of dynamic routing much less viable in Agent scenarios. Faced with this dilemma, the industry has not blindly pursued fully autonomous dynamic routing. Instead, depending on different trade-offs and priorities, three distinct engineering approaches have emerged:
The first is the Best Fixed Model approach. This is the choice of the vast majority of production-grade Agents today. The safest strategy is often to minimize mid-session switches and keep the main session locked to a single strongest model from start to finish. While this incurs higher token costs, it preserves continuous Prompt Cache hits and clear state boundaries, completely avoiding the uncertainty and engineering overhead introduced by mid-stream model switching.
The second is the Subagent Pattern. Cursor and Claude Code have adopted this approach: instead of dynamically switching models mid-session in the main conversation, offload work that can be handled independently to a subagent with a clean context. It does not require the new model to take over the entire main session — instead, it carves out a clearly defined subtask and passes along the necessary information through a structured handoff. Multi-model collaboration still takes place, but the state boundary is sharply drawn at the architectural level.
The third is the Stateful Router approach. When the task path cannot be pre-determined and the timing of switches must be decided at runtime, the Router itself needs to maintain state. SAAR (Session-Aware Agentic Routing), released by vLLM Semantic Router, attempts to persist the current model and task phase at the routing layer: lock the current model while a tool invocation is still in progress; before actually switching, check whether the state can be transferred and whether the cost savings justify the cold start.
These diverging paths reflect the compromises and constraints the application layer faces when confronting routing bottlenecks. After surveying the current industry landscape, a natural question arises: why would an open-source inference engine like vLLM — originally focused on single-node physical acceleration — cross into new territory by building a Semantic Router and even proposing the Mixture-of-Models (MoM) concept?
The answer lies in the physical-position advantage of its layer in the stack. An application-layer gateway can only see prompt semantics — it cannot see the underlying GPU KV cache distribution or cluster queuing conditions. vLLM, sitting at the inference serving layer, can access physical cache state: if the large model’s prefix cache is already resident in VRAM, continuing with the large model may be cheaper than switching to a small model. Moreover, pure single-node inference acceleration is prone to commoditized competition. By extending into cluster scheduling (Production Stack) and the routing control layer (Semantic Router), vLLM is attempting to combine model selection, safety rules, and session continuity into a unified Logical Model abstraction — thereby avoiding being reduced to a commoditized pipe by upper layers.
But no matter how the architecture evolves, it ultimately returns to the baseline question. As the vLLM Semantic Router team stated when discussing multi-model systems: a multi-model system must outperform the Best Fixed Model under approximately the same task, budget, number of calls, and latency constraints.
The complexity introduced by state management, migration, and tool compatibility must all be factored in — scores cannot be inflated by adding extra calls and retries. If using a single fixed strong model yields better total cost and success rate, then the fixed model should be chosen. For isolated, history-independent requests, selecting models by difficulty still works. But when the workload involves agentic tool-calling chains and in-progress artifacts, the problem changes: it is no longer just about which model this message should go to, but whether switching is safe right now, how state should be handed off, and whether spawning a sub-task would be a better fit.
looks good, commit it is four English words. The Router
reads the accumulated context and prior decisions, and chooses among
keeping the current model, locking the phase, or spawning a sub-task. To
choose a model for the Agent, we end up building another agent. And
before investing further complexity, determining whether the net benefit
exceeds that of the Best Fixed Model remains the most fundamental
baseline.