Stop Using Voice Agents as Dictation Tools: Deconstructing Task Contracts and Engineering Limits in Streaming Speech Recognition

From open-source Whisper to various transcription tools built upon it (such as MacWhisper, Flow, and VoiceFlow, which we developed based on the VoiceFlowKit core), voice input has become a part of daily productivity for many developers and writers. Pressing a shortcut key, speaking a passage naturally, and waiting for text to drop into the clipboard has long been a familiar interaction habit.

However, with OpenAI recently officially launching gpt-live-transcribe ($0.017/min) and gpt-transcribe ($0.0045/min) via their official announcement, alongside xAI launching the Grok STT API (see xAI developer documentation), voice API selection has once again sparked heated discussion. Many friends building products are curious: what does using this new round of dedicated streaming transcription tools actually feel like in real products?

To thoroughly dissect this round of API evolution, we can look through the three questions people care about most: * How does using this new batch of dedicated streaming transcription APIs differ from before? * Why did the model always instinctively “jump the gun” and answer when using an all-powerful Voice Agent (such as gpt-realtime-2.1) as a dictation tool in the past? * Why does “streaming words onto the screen while speaking” actually degrade the user experience? What underlying engineering bottlenecks is streaming ASR stuck on?

Combining the pitfalls we encountered and UX trade-offs we made in actual VoiceFlow development, these three questions have a clear causal logic.

Hands-on Experience with New Tools and Model Contract Conflicts

Many who have used AI voice input methods like Typeless have encountered an awkward experience: dictating a long passage of text, only for the transcribed result to feel “unauthorizedly polished.” In most cases, what users need is simple: filter out meaningless crutch words and filler words like “um” and “uh” while preserving the original expression verbatim; rather than having AI act as an editor that unilaterally rewrites phrasing or even directly answers rhetorical questions asked during dictation as commands. This “overstepping rewrite” and “jumping the gun” are precisely the biggest user experience flaws of many voice products wrapped around general-purpose LLMs.

When developing VoiceFlow, the product positioning originated from this exact pain point: it should be a pure, obedient voice input core that performs faithful transcription only—filtering noise while strictly avoiding answering questions, unsolicited polishing, or rewriting intent—and dropping clean text straight into the clipboard as soon as transcription completes.

However, in early testing when directly integrating native Speech-to-Speech foundation models like gpt-realtime-2.1 for voice input, the primary impression was not speed, but an unstoppable sense of “overstepping boundaries.” Even after repeatedly emphasizing in the System Prompt that “you are merely a transcriber, please only output what the user said,” the model would still suddenly blurt out “Okay, I understand” during pauses in speech, or outright answer questions casually asked in the audio, sometimes even repeating words or omitting sentences.

After trying a few times, one realizes that this is not a question of prompt engineering quality at all, but is determined by the underlying conditional probability mechanism of the model. When predicting the next token, Speech-to-Speech foundation models are constantly solving for such a probability:

P(Next Token | Audio, Prompt, History, Reasoning)

Under this generative objective, writing “perform transcription only” in the Prompt is at best a soft tonal hint. As long as acoustic features, pause intonations, or sentence structures in the dictation carry even a slight conversational flavor, the internal pathways for generating replies and polishing within the model will naturally activate.

The most direct change brought by Dedicated STT APIs like gpt-live-transcribe is actually “contract isolation” at the protocol level. In the session architecture for dedicated transcription, the API has no channels designed for Assistant Response, Tool Call, or Audio Reply at all, retaining only state events like transcript delta and completed. Structurally stripped of the capability to “speak back and rewrite,” the model physically cuts off “jumping the gun” and “overstepping” at the root, delivering developers a stable, boundary-clear dedicated dictation experience.

The Counter-Intuitive UX of On-Screen Streaming Text and VoiceFlow’s Design Choices

Once the overstepping issue is resolved, another intuition people naturally fall into is pursuing real-time streaming text refreshes on screen. In the early days of VoiceFlow, we also thought that “streaming words on screen while speaking” looked very cool, so we specifically tested a scheme that streamed remote-decoded Partial text onto the UI in real time via WebSocket. But once we tested it on real devices, we discovered this approach hit three major problems.

The first problem is the hidden risk to punctuation and accuracy. Streaming token emission requires the model to output words before a full sentence has been spoken. Without the support of right-context, the model cannot accurately resolve homophones and sentence-ending punctuation the moment syllables land. Text on screen is forced to constantly flicker, replace, and auto-correct right under the user’s eyes as subsequent audio flows in. According to ISCA research on Partial stability, the higher the final endpoint recognition accuracy of a model, the more severe the correction jitter during the Partial stage tends to be.

The second problem is attention fragmentation. The constantly jumping and self-correcting draft text on screen creates visual interference. While speaking, one’s attention unconsciously shifts toward the changing draft on screen, staring at uncorrected typos, which breaks the coherent flow of dictation thought.

The third problem is physical cost. To generate incremental partial tokens in real time that will be overwritten just hundreds of milliseconds later, the system continuously consumes expensive compute. High-confidence text from downstream audio directly overwrites or discards the drafts popped onto the screen within hundreds of milliseconds, resulting in wasted capital and compute.

Having stumbled through these pitfalls, VoiceFlow made a clear UX trade-off in design: completely abandon the practice of streaming draft text on screen during speech, adopting instead a safer “two-stage strategy.” While speaking, the app quietly and efficiently captures audio and maintains a persistent connection in the background, leaving the UI completely still without creating any distraction for thought. Only when Stop is pressed or a Commit signal is sent does the system initiate Finalize, smoothly rendering the final output like a typewriter in sub-second speed and placing it onto the clipboard.

The Four Engineering Challenges Behind Streaming ASR

Why do draft words jitter on screen? Why is punctuation mispredicted? Behind this lies the physical engineering laws of streaming ASR. Streaming transcription is by no means simply shoving an offline Whisper model behind a WebSocket, but rather a re-architected acoustic and language decoding pipeline under strict time budget constraints:

Microphone PCM Sampling -> Chunk Aggregation (100ms) -> Causal Encoder (Look-Ahead) -> Monotonic Alignment & Draft Stream -> Endpointing Smart Turn -> Final Output

1. Chunk Aggregation and Look-Ahead “Peeking into the Future”

The audio front-end first slices the continuous waveform into analysis frames of 10 to 25 milliseconds, which are then further aggregated into Chunks of around 100 milliseconds for transmission or inference. Slicing into smaller Chunks triggers network and compute earlier, but exponentially increases message queuing and state overhead; larger Chunks provide richer local acoustic evidence, yet naturally add waiting latency.

A more critical bottleneck lies in right-context. When a syllable is spoken, looking solely at the acoustic features at hand makes it impossible for the model to distinguish homophones or determine whether the current syllable marks the end of a sentence. Zero-latency emission that completely refuses to wait for future audio inevitably leads to errors in homophones and clause-ending punctuation.

In practical engineering, streaming encoders typically need to introduce a Look-Ahead mechanism of 100 to 300 milliseconds—allowing the model to “peek” at a short snippet of future audio. These few hundred milliseconds of Look-Ahead are a physical necessity to avoid misrecognized words; only by providing the decoder with a small amount of right-side acoustic evidence can punctuation and word semantics remain stable.

2. Causal Encoder and Memory Bank

When performing offline transcription, Whisper can comfortably run omnidirectional bidirectional Self-Attention over the entire recording to examine all surrounding context; however, in streaming scenarios, the encoder simply cannot wait for future audio and is forced to perform unidirectional Causal Encoder encoding.

If the system were to recompute features for all preceding historical audio every time a new 100-millisecond Chunk arrived, computational complexity would explode quadratically with audio length, instantly overwhelming real-time compute budgets.

To solve this challenge, modern streaming architectures (such as the Emformer architecture) introduce KV Cache and Memory Bank mechanisms. Memory Bank acts like historical memory cards compressed and saved over time, allowing the encoder to focus solely on computing the current Chunk and retrieving historical memory, avoiding redundant computation over the previous minutes of audio.

3. Monotonic Alignment Token Emission and Mutable Partial Incremental Correction

In mapping acoustic features to text, the underlying graph of streaming ASR requires that the output sequence must emit tokens monotonically forward over time, unlike LLMs which can jump back to rewrite previous text.

The classic CTC algorithm outputs text Labels or Blank symbols independently at each frame, boasting extremely fast decoding speed, but because frames are mutually independent, it lacks the context correction capabilities of a language model; meanwhile, Transducer (RNN-T) integrates a language prediction network, enabling it to utilize generated text history to correct homophone errors while maintaining forward monotonic token emission over time. The FastEmit algorithm adds a penalty term for blank frame delay into the loss function, forcing the model to emit tokens as early as possible.

However, to achieve greater flexibility between monotonic emission and recognition accuracy, modern streaming APIs (such as gpt-live-transcribe and Grok STT) introduce the concept of Mutable Partial (modifiable draft stream) at the transport layer. Before receiving the Committed Final endpoint signal, the decoder is allowed to make local fine-adjustments to earlier emitted draft text within the same sentence (such as modifying near-homophones or adding punctuation) as new audio Chunks arrive. This requires the client not to mechanically append characters, but to maintain a UI state machine by Item ID to smoothly overwrite the draft area.

4. Endpointing Smart Turn: Automatic “Finished Speaking” Determination Without Manual Intervention

Compared to the first three internal acoustic and decoding hurdles of the model, the fourth is a genuinely high-difficulty systems engineering challenge: without the user manually clicking “I’m finished speaking,” how does the machine fully automatically determine when the user has ended a sentence or an intent?

In continuous dictation or voice interaction, users rarely hit the stop key manually on a frequent basis. The Endpointing system must conduct a real-time background trade-off: if segmentation is too aggressive (for example, truncating as soon as 200 milliseconds of silence is detected), the aggressive segmentation will split normal thinking pauses into fragmented sentences; if determination is too conservative (waiting for silence over 1 second), users are left waiting helplessly in front of a still screen after finishing speaking.

Research shows that silence-waiting overhead in Endpointing accounts for over half of total perceived user latency. Merely increasing the decoding speed of the model itself will be imperceptible to users if conservative silence thresholds are kept. Smart Turn or Semantic VAD combining semantic completeness attempts to navigate the trade-off between acoustic silence and semantic integrity, but this remains the most challenging systematic engineering obstacle to overcome in real-time ASR.

Model Selection in Practice and Conclusion

In practical deployment and model comparison, performance and cost-effectiveness across models exhibit stark differences.

Taking OpenAI’s technology stack as an example, the legacy gpt-4o-transcribe performed poorly in practical product testing, easily dropping words or missing phrases in long sentences; whereas the newly launched gpt-transcribe ($0.0045/min, roughly $0.27/hour) fully reaches gpt-realtime-2.1-level transcription fidelity. Furthermore, because the Response channel has been stripped away, its output is significantly more stable, and its price is roughly one-fifteenth that of a Realtime Voice Agent.

By contrast, xAI’s Grok STT API (Batch mode $0.10/hour, Streaming mode $0.20/hour) trails OpenAI’s top models slightly in accuracy under complex edge cases, but its performance comes extremely close. Its core strengths lie in speed and cost: processing response speed is roughly 10x faster than OpenAI’s models, while total cost is only a tenth.

For this reason, VoiceFlow implements a decoupled persistent configuration in its underlying model strategy: users can use high-responsiveness Realtime / Streaming protocols during real-time dictation, while switching at any time to Grok Batch mode or gpt-transcribe for long-text processing that balances extreme response speed and high cost-effectiveness.

As of July 31, 2026, neither OpenAI (gpt-live-transcribe / gpt-transcribe) nor xAI (Grok STT) has released an independent Technical Report or underlying neural network architecture documentation for their new STT models. Official accuracy and WER figures remain vendor-reported data; third-party evaluations like Artificial Analysis’s independent STT benchmark have also primarily verified Batch mode so far. This serves as a reminder when choosing models: do not rely solely on official benchmark numbers—be sure to run real tests with long-tail audio from your actual usage scenarios.

From blindly chasing “voice LLMs doing everything” and “on-screen real-time word streaming” early on to returning today to an independent Dedicated STT contract, this represents a rational cognitive reset: the true quality of a voice transcription user experience has never been about popping text onto the screen a few milliseconds faster than others, but whether the task contract is clean enough, whether trade-offs around right-context are reasonable enough, and whether the user’s thought process during dictation is sufficiently respected.