Over a four-day span last week, three companies released real-time voice products or research previews. On October 20, Mistral launched the Voxtral 3 voice agent toolkit, and ElevenLabs released Eleven Realtime. On October 23, Block introduced its research model Dessa. Although all three are billed as real-time voice products and deliver fluid conversations in demos, they work very differently under the hood.
Understanding how these products relate to one another comes down to asking two questions—and then checking whether they add a layer of delegation that splits immediate conversation and slower task execution between a frontend model and a backend model. Those two questions let us place the major real-time voice products on the market into a single framework. Instead of poring over model leaderboards when choosing a stack, engineering teams can first ask what their business cannot compromise on, and then work backward along this framework to pick an architecture.
Two everyday analogies help make these two choices concrete. The first contrasts passing notes with simultaneous interpretation. Passing notes is three people sitting at a long table dividing up the work: the first person listens to the customer, writes down what was said on a slip of paper, and hands it to the second person; the second person looks up the needed information from the note, writes out a reply, and passes it to the third person; the third person reads that note into a microphone. Simultaneous interpretation is a single interpreter sitting alone in a soundproof booth, listening to a foreign language through headphones and speaking the native translation almost at the same moment—one person listening and speaking at once. In voice systems, this contrast maps to the first choice: whether to split the system into separate components or use a single model.
The second contrast is between talking on walkie-talkies and chatting at the dinner table. On walkie-talkies, two people speak one turn at a time: Speaker A finishes a thought and releases the button, the radio beeps, and Speaker B hears the beep before pressing the button to reply. Two people chatting at the dinner table look very different: while Speaker A tells a story, Speaker B listens, nods, murmurs agreement, and chimes in naturally halfway through. This contrast maps to the second choice: who decides turn-taking. In one approach, a component outside the model listens for pauses and checks semantics to decide on the model’s behalf whether the user has finished speaking. In the other, the model listens and speaks at the same time, deciding on its own when to speak up, when to pause, and when to interject. This mode—where the model listens and speaks simultaneously and manages its own timing—is commonly called full-duplex.
Lining up these two choices gives us a two-level decision tree. The first level asks whether we split the system into multiple components or use a single model. Splitting into multiple components gives a cascade: the system is divided into speech-to-text, a large language model (LLM), and text-to-speech, passing text between stages. Mistral’s Voxtral 3 chains three independent models into a pipeline, allowing developers to swap out the LLM or the voice independently. And although ElevenLabs markets Eleven Realtime as a speech-to-speech model, ElevenLabs’ compatibility guide FAQ states that it is also a cascade internally, with the components packaged together by the vendor—a reminder that you cannot tell a product’s architecture from its name alone. In a cascade, deciding when the user has finished speaking must happen in a component outside the LLM, such as the transcription model. So this branch ends here.
Along the single-model branch, the second choice is whether turn-taking is decided externally or by the model itself. External turn detection gives a turn-based architecture: Alibaba’s Qwen3-Omni is typically used this way, and the open-source inference framework SGLang groups it into this category in the official SGLang blog. Having the model decide turns on its own gives native full-duplex, exemplified by Kyutai’s Moshi, NVIDIA’s PersonaPlex built on the same architecture, OpenBMB’s MiniCPM-o 4.5, and Block’s Dessa. As SGLang describes it, turn-based sits between native full-duplex and a cascade: a single model integrates multimodal perception and speech generation, while when to speak and when to interrupt are configured externally.
On top of a native full-duplex frontend, you can stack a layer of delegation—what we call native frontend plus delegation. OpenAI’s GPT-Live follows this pattern: the frontend model decides multiple times per second whether to speak, keep listening, or interrupt, and hands off slow tasks to a backend. Cascade, turn-based, native full-duplex, and native frontend plus delegation—these four names read directly off the tree, and we will use them consistently throughout the rest of this article.
The question marks in the diagram indicate details not disclosed officially. GPT-Live’s API outputs transcripts and text responses, but OpenAI has not stated whether the model also generates text before audio internally. ElevenLabs has not specified which stage in Eleven Realtime’s cascade decides when the user has finished speaking, nor does it describe the product as full-duplex. For Dessa, Block has so far released only an announcement, 21 curated audio clips, and a waitlist, without disclosing how it works internally or whether it supports tool calling.
Once we see the branches on the decision tree, we can examine the cost behind each one. The first set of trade-offs comes from real-time responsiveness. Imagine someone standing across a ping-pong table firing a ball at you every half second, expecting you to return each shot on the spot. While you are swinging your paddle, someone else hands you a complex tax form and asks you to calculate your tax bill for the year in your head. As long as you have to keep returning the ball in front of you, you cannot freeze for ten seconds to do tax math. Real-time voice conversation faces that constantly flying ping-pong ball. Because long pauses make human conversation awkward, a frontline voice model must respond with audio in a fraction of a second. To compute fast enough, the model cannot be too large, nor can it run long multi-step reasoning at each step—which naturally caps its intelligence. The official GPT-Live system card confirms this: both GPT-Live variants, GPT-Live-1 and GPT-Live-1 mini, trail GPT-5.5 Thinking across multiple intelligence benchmarks. Without a backend model, GPT-Live can barely call tools on its own and cannot execute code.
Real-time responsiveness also shows up directly on the bill. A real-time voice session is billed very differently from a standard text model. A text model only consumes compute during the few seconds it takes to generate a reply; when the user sits staring at the screen thinking, it costs nothing. A voice session, by contrast, ties up compute from the moment the call connects until it hangs up. Even when both sides are silent, the system is continuously listening and evaluating, so billing runs by call duration. According to the GPT-Live-1 pricing documentation, the frontend voice layer costs $0.05 per minute, billed by the second—about $3 per hour, with backend models and tools billed separately. By comparison, one-way text-to-speech with Gemini 3.8 Flash TTS, which does not support real-time conversation or function calling, costs roughly $0.81 per hour of audio output during its promotional period. Two-way real-time sessions cost several times more because what you are paying for is that always-on real-time loop.
The second set of trade-offs revolves around text. Across the decision tree, reliance on text fades step by step. The cascade branch relies on text from start to finish. On the single-model branch, models such as Qwen3-Omni, Moshi, PersonaPlex, and MiniCPM-o 4.5 still maintain an internal text stream, effectively writing text in their heads as they speak. Further along, Block claims that Dessa is trained exclusively on recordings of human conversations, without using text at all.
The more text in the loop, the more controllable the system. Customer support teams in banking or e-commerce dread mishearing a refund amount or being unable to audit what went wrong afterward. Because a cascade has visible text in the middle, there is a plain-text record of what the user said and which tools the model is about to call. Engineering teams can inspect logs, apply filtering rules before text reaches the LLM, and swap out individual components that are too expensive or not capable enough. The cost is that hesitation, sighs, and sarcasm in the speaker’s tone are mostly lost the moment speech is flattened into text, and converting audio to text and back to audio adds another layer of latency. Conversely, the further you move toward native full-duplex—where the model listens to audio and replies with audio directly—the more natural the conversation becomes: the model picks up on tone of voice and chimes in right as the other person catches their breath. Yet the intermediate steps become harder to inspect, tools become harder to wire up, and compliance logging requires bolting on external transcription anyway.
There is also a trade-off inside the second choice itself: who decides turn-taking affects noise robustness and interruption handling. Even looking strictly at background chatter and interruptions, each branch on the tree has its own weaknesses. The SGLang team ran a small-sample official benchmark comparing native full-duplex (represented by MiniCPM-o 4.5), turn-based, and cascade systems on noise resistance and interruption handling. With background chatter present, native full-duplex was better at ignoring distractions and continuing to speak. When the user genuinely interrupted to make the model stop, turn-based and cascade systems stopped faster, but they were also more prone to mistaking a bystander’s remark for a new user turn. Native full-duplex was slower to stop and had a lower response rate to genuine interruptions. With only 10 to 25 test cases per group, the sample size is small and directional only, leading the authors to conclude that no single approach wins across the board.
If a frontline voice model cannot pause to think deeply while returning ping-pong balls, what happens when it runs into heavy, complex work? Imagine you are driving and call a travel agency to rebook a connecting flight. The moment the call connects, you want the agent to respond right away: “Got it, let me check seat availability for this afternoon right now—drive safely.” Over the next several dozen seconds, the agent looks up airline change policies in the background, checks seat inventory, calculates the fare difference, and finally reads the options back to you. Keeping the conversation going demands sub-second responsiveness, whereas rebooking the ticket demands tens of seconds of deep reasoning and system calls—requirements for speed and intelligence that pull in opposite directions. Cram both jobs into a single model and the line goes dead the moment the model starts thinking; force a lightweight model to do the math and it will routinely miscalculate fare differences or call the wrong APIs.
The delegation layer exists to resolve this tension between fast and slow. It slices the system along a different axis than a cascade. A cascade slices by processing stage—dividing work into a three-stage pipeline of listening, thinking, and speaking that every utterance must traverse in sequence. Delegation slices by timescale—separating a fast loop that keeps the conversation flowing from a slow loop that gets the work done. An agile full-duplex voice model handles the conversation up front, while a reasoning-heavy LLM or agent tackles the heavy lifting in the background.
This delegation architecture is what is new about GPT-Live. Full-duplex speech has been around since Moshi launched in 2024; GPT-Live’s contribution is pairing a full-duplex frontend with a slow-thinking backend. The frontend keeps the conversation going, hands off search, reasoning, or tool execution to the backend, continues chatting while the backend works, and weaves the result back into the conversation once it is ready. With the launch of the GPT-Live API release page this September, the delegation API is now officially available. Standing between the frontend and the backend is your application. In the flight-rebooking example, three roles coordinate to complete the job: the frontend voice model handles the live conversation and decides when to ask the backend for help; the backend model or agent figures out how to make the change, such as checking fare rules and calculating price differences; and anything that touches your internal systems—such as checking seat inventory or actually modifying the booking—is executed via functions in your application, which also handles permission checks, user confirmation, and saving task progress.
As noted in OpenAI’s live delegation documentation, you choose between two delegation modes when creating a session, and cannot switch mid-session. In managed mode, OpenAI runs the backend model while custom functions still execute inside your own application. Client delegation lets you plug in any model or agent you run yourself. A live transcript of the conversation is streamed to your application throughout; in client delegation, the delegation signal emitted by the frontend simply indicates that it is time to hand off work without carrying the verbatim transcript, so your application packages the task from the streamed transcript and its own state before sending it to the backend. Once the backend finishes computing, it returns the result to the session, with the option to either read it aloud or inject it silently into the context. If the user interrupts, the backend can keep running, leaving your application to decide whether to let it finish or cancel the job.
This handoff mechanism also explains where text fits in. Although text drops acoustic information from speech, it serves as the lingua franca for the frontend to delegate tasks to the backend and for the backend to return results—and the session context itself is carried in text. Consequently, we suspect that the more native a model is and the less it uses text, the more natural its frontend becomes, but the harder it is to wire up a backend. For example, Block has not yet disclosed how Dessa—which uses no text during training—receives context or takes back results from a backend. We suspect backend results must either be converted into audio before being fed to Dessa, or passed in through a separate text channel bolted onto the model.
Once you see the decision tree clearly, the next step before building a system is to separate capabilities that are shifting rapidly over a six-month horizon from those that remain stable over the long term.
Three things are changing rapidly. First, native models are starting to call tools on their own. For example, NVIDIA’s September Nemotron VoiceChat paper demonstrated an open full-duplex voice model that decides which tools to call by itself, without first transcribing speech to text and handing it off to another model. Second, pricing: one-way speech synthesis has already fallen below one dollar per hour, and we expect the cost of two-way real-time sessions to keep dropping as well. Third, end-of-turn detection is moving from simply listening for silence to combining acoustic cues with semantics. Voxtral 3 builds this directly into its transcription model—though this remains an improvement within a cascade, where turn-taking is still decided by a component outside the LLM.
At the same time, three things will remain constant over the long run. First, regardless of architecture, users will interrupt mid-sentence, and the system must halt unplayed audio and truncate the conversation history to match what the user actually heard. Second, the need to hand off heavy lifting to a stronger backend: as long as humans expect conversational replies within a few hundred milliseconds, the division of labor between fast and slow loops is here to stay. Third, enterprise requirements for audit trails, permission checks, and reliable tool calling.
Understanding both sides lets us make the right engineering trade-offs: keep the fast-changing parts swappable, and build the unchanging parts into your own infrastructure.
With the trade-offs and the shifting versus stable pieces clear, choosing a stack does not mean hunting for the highest overall score on a leaderboard; it means returning to your use case and asking what you cannot afford to sacrifice. A customer calling support for a refund and a young person wide awake at midnight chatting with a virtual companion have diametrically opposed definitions of “good.” The caller seeking a refund can tolerate an extra half-second pause before each reply, but cannot accept a wrong refund amount or an agent that promises a refund without actually filing the ticket. The person chatting with a virtual companion does not care whether it can query a database, but will close the app after two exchanges if it sounds like it is reading from a script. Different non-negotiables naturally lead to different branches when working backward along the decision tree.
| If you cannot afford to sacrifice | Right approach | Why |
|---|---|---|
| Getting the job done, full auditability, predictable cost | Cascade | Visible text in the middle makes it inspectable, lets you add rules, and lets you swap components independently |
| Letting the model hear tone directly while you control when to speak and interrupt | Turn-based | A single model handles perception and speech generation, while turn-taking is configured externally |
| Natural speech, emotional tone, spontaneous backchanneling and interjections | Native full-duplex | The model decides turn-taking on its own |
| Both natural conversation and complex heavy lifting | Native frontend plus delegation | The frontend keeps the conversation flowing while the backend does the heavy work |
For dictation, however, you should not use a conversational voice agent at all. As we analyzed in our July article on dictation, conversational agents inevitably try to answer or rewrite what you say. A dedicated transcription service simply turns audio into text without chiming in.
Once you pick a branch, the next decision is between managed and self-hosted deployment. A fully managed stack saves operational effort, but the platform draws the boundaries of what you can customize. For instance, according to ElevenLabs’ realtime model configuration documentation, the components inside Eleven Realtime are chosen by the platform, so you cannot swap in your own LLM in Realtime mode. If data cannot leave your environment, self-hosting open-weight models is the way to go—whether as a cascade built from open-source components or with native full-duplex models such as Moshi, PersonaPlex, or the Apache-2.0-licensed MiniCPM-o 4.5.
When we selected architectures for two of our own voice entry points, they landed on two different branches. The first is a mobile voice assistant where users casually hand off tasks while chatting and let the backend do the work; what we could not sacrifice was natural conversation paired with heavy task execution, so we chose native frontend plus delegation. We run GPT-Live as the frontend in managed mode, where the backend model decides which functions to call; because those functions still execute inside the application in managed mode, we run them on the phone to operate our own self-hosted agent service. The second is a local desktop microphone entry point whose only job is to understand commands and dispatch tasks reliably, with no small talk required; there we chose a cascade so the intermediate text stays visible and can be validated before execution. The same team picked two different architectures because what we could not afford to sacrifice was different in each case. Once the architecture is set, the next question is which layer actually owns the conversation state—a topic we will pick up in our next article.