Over the past two years, building applications or agents on top of language model APIs rarely required worrying about where conversation state should live. The standard practice was to send stateless requests: the application carried the entire conversation history, and the server discarded the state right after computation. Under that model, switching providers meant simply changing an endpoint URL, and load balancing was straightforward. Now things have changed. Because many agents run time-consuming background tasks and voice conversations must support interruption at any moment, more and more state is moving to the server side. In this world, starting a request is easy, but canceling one mid-flight is much harder. For example, if a user changes their mind halfway through while a previously triggered background task is still running, and the application fails to stop it and discard its output, the late result will bleed into the next turn of conversation and be spoken aloud.
In March 2025, OpenAI launched the agent-oriented Responses API while continuing to support Chat Completions. According to OpenAI’s conversation state documentation, responses from the new API are stored on the server by default and can be reused across turns. Once conversation history is no longer carried with every request, whoever hosts that state determines the future cost of switching engines. If you deploy models yourself or provide inference serving, this directly involves the inference engine layer: whether the engine should manage session state. Even if you only call APIs, the inference engine behind your provider is making the same choice, which shapes the session interface you get and what you have to migrate when switching providers. Real-time voice is where this problem surfaced first. This October, the open-source inference engine SGLang offered its answer. Let’s step into the designers’ shoes to see what problems it faced and what decisions it made.
Text chat works one message at a time: you send a line and wait for a reply. Real-time voice can be interrupted at any moment. As soon as the user cuts in mid-sentence, what the server has computed diverges from what the user actually heard, and the stateless model of sending the full history with every request breaks down first.
For example, suppose you are talking to a voice assistant on your phone through headphones. It plans to say three sentences, and halfway through the second sentence you suddenly jump in with another question. That unfinished utterance now exists in three places at once: on the remote GPU, the model has already computed the third sentence; the server has just sent out the audio packets for the second sentence; but in your headphones, you have only heard the first sentence plus a few words. The moment you interrupt, the system must immediately halt model computation, stop the audio still in transit over the network, and trim the conversation history back to the exact point the user actually heard. When the next turn begins, the assistant should only remember the half-utterance that reached the user’s ears. As SGLang’s official real-time runtime blog post points out, just because the server has generated or sent audio does not mean that audio has been played back—actual playback progress can only come from client feedback.
Handling interruptions requires the system to maintain two kinds of state: one inside the inference engine, tracking how far model computation has progressed and whether it can stop immediately; and another on the client, tracking how far headphone playback has actually reached. Regardless of who manages the session, someone must keep these two states consistent. SGLang’s alignment implementation is designed specifically for this purpose.
From the perspective of an inference engine designer, another challenge is that real-time voice currently has no single unified approach. Four approaches exist on the market: the first is a cascade of three separate components—speech recognition, a language model, and speech synthesis; the second is single-model turn-based interaction, where a user’s spoken segment is fed as context to a multimodal model; the third is native full-duplex, where a single model processes a continuous audio stream and decides on its own when to listen and when to speak; and the fourth is a native frontend with delegation, where a frontend voice model decides when to listen and speak while conversing with the user in real time, and backend large models and tools handle time-consuming reasoning and tool calls.
These four approaches place conversation state in different locations. In a cascade, conversation state lives primarily in the external orchestrator—the application layer that chains together recognition, generation, and playback—and the state itself is a text history. In a turn-based setup, conversation state lives in a single model’s session, while an external component determines when an utterance has ended. In native full-duplex, conversation state lives in the continuously advancing model session on the GPU. In a native frontend with delegation, conversation state spans two paths running at different speeds: the fast frontend voice session and the slower background tasks. For a detailed comparison of the four approaches, see our companion article on real-time voice architectures. Each approach has its own drawbacks. The capabilities required for the fourth approach are not yet complete in SGLang; SGLang officially evaluated only the first three and concluded that no single approach is best across all scenarios.
At the application layer, open-source frameworks already perform playback alignment. LiveKit and Pipecat are two widely used open-source frameworks for real-time voice applications that routinely handle turn management, interruption handling, and media transport. LiveKit’s code for trimming history by playback position writes only the text that was actually played out in sync into the conversation context once playback ends. Pipecat’s playback feedback code similarly uses timestamps to append corresponding text to the conversation history only after the audio has actually been played. Trimming history by playback position is therefore not a SGLang invention. The real dilemma facing SGLang’s designers was whether, given these four coexisting architectures, managing an entire conversation should remain in the application layer or be pulled into the inference engine.
Whether to put conversation state inside the inference engine depends on what the designers are trying to achieve. SGLang’s designers wanted the system to meet three goals at once. First, the upper-layer interface should be independent of the underlying model and architecture, so developers do not have to rewrite their interaction logic when swapping models. Second, when a user interrupts a conversation, the cancellation signal must actually reach the place where computation happens. Third, different real-time voice architectures should be directly comparable under the same benchmark standard.
Driven by these goals, SGLang’s designers made two design decisions, backed by a core premise: interaction rules evolve much more slowly than model architectures. The first decision was to fix the interaction rules as the interface exposed to upper layers, making the underlying models and components swappable. How a turn is defined, what gets canceled upon interruption, and how conversation history is truncated to match what the user heard are fixed rules for the upper layer. Beneath this interface, developers can upgrade the language model in a cascade without touching speech recognition or speech synthesis, or swap out only turn detection while leaving other components intact, reusing all the surrounding session infrastructure.
The second decision was to change the basic unit of the inference engine from a single request to a persistent session. A session spans the entire conversation from start to finish; finishing a single response does not close the session, and canceling in-progress work mid-turn does not discard the context needed for the next turn.
Making this second decision required giving up many benefits of stateless requests. Stateless requests are independent of one another, different inference engines can be swapped freely, request routing and load balancing are simple, and if one machine goes down, a request can be retried on another instance. Once session state resides permanently inside the inference engine, those advantages are forfeited, and migrating an application to a different underlying inference engine becomes much more costly. Moreover, SGLang does not yet support session resumption after a disconnect—reconnecting after a network drop to continue chatting in the same session. Once the connection drops, the session state on the server can no longer be reattached.
This design had been in planning since early in the year, starting with a January design proposal and culminating in the October 20 release of SGLang’s unified interactive runtime. What this runtime unifies is interaction control and evaluation; it does not turn different architectures into the same underlying computation.
The first decision applies to all four approaches, because every approach needs to define what counts as a turn, what gets canceled on interruption, and how much the user has heard. Here we examine the second decision and ask whether the session actually needs to live inside the inference engine. Which approach you use determines whether you need an inference engine that holds onto sessions.
Take native full-duplex first. Some models process a frame of audio every 80 milliseconds, keeping conversation state continuously inside GPU memory where external applications cannot see it. To make the model stop speaking, cancellation must take effect inside the inference engine. Through SGLang’s control implementation, a stop signal can be delivered directly to the computation currently running in the engine. With this approach, putting the session inside the inference engine is almost mandatory.
Single-model turn-based interaction sits halfway inside the engine and halfway outside. Qwen3-Omni, the multimodal model from Alibaba’s Qwen team, runs in this mode in SGLang. Stopping model generation immediately has to reach inside the engine, whereas determining whether the user has finished speaking and how much they actually heard can just as well be handled by an external framework.
For a cascade, the main reason to put the conversation inside the inference engine is the convenience of unifying interfaces and evaluation across all four approaches. Figuring out how much the user heard requires only text, audio, and a client truncation signal; it needs no internal information from the inference engine and can be implemented in the application layer. SGLang’s original paper introduced RadixAttention for prefix caching, where repeated history is computed once and cached, making it inexpensive to resend history every turn and weakening the case for keeping cascade state resident on the server. Furthermore, recognition, generation, and synthesis in a cascade are often separate services. In SGLang’s cascade component implementation, when SGLang cancels a request, the user’s headphones go quiet right away, but the remote model service may not stop computing immediately—when it stops depends on how it handles disconnects and aborts. Because the cancellation signal cannot reach the place where computation happens, the second goal cannot be guaranteed here, making engine-managed sessions for cascades primarily a matter of convenience.
Finally, the hybrid architecture of a native frontend with delegation requires cooperation from the inference engine. The fast and slow paths share a single session, and if background computation also runs inside the engine, only the engine knows how far that computation has progressed. If this hybrid architecture becomes mainstream, the case for putting session control inside the inference engine is strongest.
Hybrid architectures especially require keeping state synchronized between the engine and external orchestration—for instance, discarding stale results upon interruption and ensuring late results are never spoken aloud. No one has finished building this yet. SGLang’s asynchronous task support is still on its real-time session roadmap. vLLM-Omni, the omni-modal project from the open-source engine vLLM, is a step ahead at the inference engine layer (see its real-time protocol documentation). OpenAI’s GPT-Live leaves this to the application to handle (see the official GPT-Live getting started guide).
In short, native full-duplex almost inevitably requires placing the session inside the inference engine; the hybrid architecture has the strongest justification if it becomes mainstream, though it is not yet complete; and putting cascades inside the engine is mostly for convenience. Inference engines usually compete on raw speed and day-zero support for new models. SGLang did not have a clear edge on either front, so it placed its bet on the session layer.
There are roughly three reasons SGLang moved up the stack into session management. First is its academic pedigree: SGLang’s original paper argued that an engine must understand program structure in order to optimize execution, and this work applies the same philosophy to multi-turn conversations. Second is its evaluation heritage: SGLang comes out of LMSYS, which built Chatbot Arena and released a unified benchmark here as well. Comparing several approaches with the same audio recordings and scoring requires them to speak the same session protocol. Third is its position in the stack: by claiming the conversation layer, the engine avoids being reduced to a commodity executor behind an OpenAI-compatible endpoint that can be swapped out at any time.
Different players have given different answers on which layer should own conversation state: vLLM-Omni has the engine provide only foundational primitives while leaving orchestration outside; the LiveKit Agents documentation and the Pipecat project place it in the application layer; native models learn when to listen and speak directly in their weights; and managed services seal everything inside. Their concrete progress also varies—for instance, vLLM-Omni already offers session resumption after disconnects for some models, whereas SGLang currently does not.
For an abstraction to be adopted, it must make the things users change most often easier. SGLang’s abstraction suits teams that frequently swap models and benchmark architectures, whereas most voice product teams spend their time changing prompts, tools, business logic, and telephony integration—all of which live in the application layer.
Some users have already wired this real-time interface into production (see one user’s account), though they had to implement billing, idle timeouts, and resource reclamation externally on their own.
At present, support for cancellation, truncation, and tool use across different models in SGLang remains uneven, and none of them support session resumption after disconnects. Once state stays resident on the server, both SGLang and vLLM-Omni also recently had to fix bugs where conversations from different users bled into one another (SGLang’s fix).
What SGLang and vLLM-Omni are competing over is which set of rules upper-layer frameworks choose to synchronize state with the inference engine. If upper-layer frameworks adopt SGLang’s session interface as a standard, its bet will have paid off. Two signals are worth watching: whether SGLang can run newly released open voice models on day one, and whether Pipecat and LiveKit integrate its session interface rather than routing around it.
For engineers who only call APIs, evaluating lock-in risk comes down to two things: whether the session interface is an open standard or packs proprietary extensions under a generic API name, and whether session state can be exported and reconstructed elsewhere. Currently, the real-time interfaces of both SGLang and vLLM-Omni add their own extensions on top of generic events, and SGLang’s benchmark client relies on those extended events. Whichever session interface becomes the standard for upper layers will determine where your conversation state ultimately lives—and how much you will have to pay to switch engines in the future.