The Token-Flush Latency: Why 'Full-Duplex' Voice AI Needs More Than Concurrent Streams

'Full-duplex voice AI conversation' is a misnomer. It promises seamless interaction, but delivers an orchestrated illusion of simultaneity. The real challenge for engineers lies in synchronizing disparate components to deliver a sub-100ms P95 latency, especially when human interruptions are the norm, not the exception. Simply upgrading ASR or TTS models won't cut it; the entire synchronization fabric needs rethinking for production systems.

The core problem I consistently encounter, whether on an industrial edge gateway or a cloud-native deployment on GPU clusters, is the data flow between discrete processing units when human interruption is not just possible, but expected and even encouraged. If I'm building a real-time voice assistant, the fundamental design question quickly becomes: how does the system listen, process, and generate its own response, all while being prepared for a user to jump in at any millisecond? Naive buffering and sequential processing introduces latency spikes pushing 800ms, effectively ruining the conversational flow.

Let's illustrate a common failure point for achieving true full-duplex:

USER -> [Microphone] -> Audio Buffer (A)
                  |
                  v
              [ASR Stream]
                  |
                  v
    Transcription Chunks (T_C)
                  |
                  v
[LLM (Stream In)] --> Processing --> [LLM (Stream Out)]
      ^                                      |
      |                                      v
    USER <---- [Speaker] <---- Audio Buffer (B)
                  ^
                  |
              [TTS Stream]

In a traditional half-duplex system, the ASR stream largely finishes before the LLM (and subsequently TTS) takes over. The user explicitly waits. For full-duplex, ASR Stream (perhaps on a dedicated DSP running a quantized ASR model) and TTS Stream operate in parallel, contending for resources and, more critically, for the state of the 7B parameter LLM. The LLM (Stream In) must constantly accept new T_C while LLM (Stream Out) continuously generates a response. This means the large language model needs to handle an incoming query's continuation or interruption while still generating its current reply.

The real headache comes down to the interruption mechanism. How do I signal to the LLM that its current output stream should be aborted or redirected because the user has started speaking again, and how quickly can that signal propagate with sub-20ms message propagation? The delay here often reveals the underlying half-duplex scaffolding. When a user interrupts, the ASR Stream registers new audio. This needs to trigger a "stop" or "re-evaluation" command to the LLM (Stream Out) and prioritize the new T_C coming into LLM (Stream In). This requires not just fast I/O but intelligent model management, such as flushing the LLM's speculative decoding buffer and invalidating the KV cache entries. This state management is expensive. I still don't fully understand why some commercial ASR services consistently add an internal 200ms buffer, even when configured for low-latency streaming, effectively negating any advantage from server-side token-by-token LLM generation.

My observation is that systems claiming "full-duplex" often implement robust VAD (like Silero VAD) and overlapping ASR/TTS, but the underlying LLM inference itself still operates in a semi-sequential, "interrupt-and-restart" mode. The naturalness suffers. The system pauses mid-sentence, processes the interruption, and then starts a new response, rather than smoothly weaving the interruption into the ongoing dialogue.

Achieving genuinely natural, full-duplex interaction at scale demands tightly coupled, real-time synchronization primitives between all these stages. The mechanism for interruption cannot be a blunt reset; it needs to be a nuanced, context-aware adaptation, minimizing discarded work while maintaining conversational flow. This implies LLM architectures designed for continuous context updates, not just batch processing.

Delivering true full-duplex voice AI to production necessitates an end-to-end design that embraces continuous, concurrent processing and immediate, context-aware state transitions, rather than sequential logic overlaid with quick interrupts.