A voice agent that takes three seconds to answer feels broken. Not slow — broken. Callers talk over it, repeat themselves, or hang up. In ordinary human conversation, the gap between turns averages around 200 milliseconds. Once a pause crosses one second, people hear hesitation or a dead line.
The only number that matters here is voice-to-voice latency: the time from the moment the caller stops speaking to the moment they hear the agent's first syllable. In TeleVox AI, the voice agent platform we built and operate, we measure roughly 0.7 seconds voice-to-voice in production. This post walks the pipeline stage by stage and shows where the milliseconds go.
The pipeline
TeleVox runs four stages: Silero VAD for speech detection, Deepgram streaming STT, an OpenAI model for reasoning, and Deepgram Aura-2 for synthesis. Transport is WebRTC via LiveKit, terminating on a fleet of stateless voice workers on Fly.io.
The naive wiring: wait for silence, transcribe the utterance, send the transcript to the LLM, wait for the full completion, synthesize the whole reply, play it. Sequential sums like that land between three and five seconds. Everything below is about breaking the sequence. Every stage streams. No stage waits for the previous one to completely finish.
Stage 1: deciding the caller is done
Turn detection is where most voice agents quietly lose their budget, because it sits before everything else and it is invisible in vendor latency claims.
The blunt approach is a silence timeout: no speech for N milliseconds means the turn is over. Set N to 700 ms and you add 700 ms of dead air to every single response before any model runs. Set it to 300 ms and you interrupt anyone who pauses mid-sentence — "my number is... 555..." becomes a fight with the agent.
We run two things together. Silero VAD executes in-process on the voice worker, classifying small audio frames as speech or silence in a few milliseconds each — it is the cheap, fast signal. On top of it we layer semantic turn detection using Deepgram's interim transcripts: "What's your address?" is a complete thought and can end the turn on a short silence, while "My address is" is obviously not, so the agent keeps listening even through a longer pause. Semantic endpointing let us pull the effective silence window down without cutting people off. It is the single highest-leverage change we made.
Stage 2: streaming STT
With Deepgram's streaming STT, transcription happens while the caller is still talking. Interim results arrive continuously, so by the time turn detection fires, nearly the whole utterance is already transcribed. The cost at turn end is just finalizing the last words — a small tail, not a full transcription pass.
This is also what makes semantic turn detection possible at all: you cannot reason about whether an utterance is complete unless you have its text before the turn officially ends.
Stage 3: LLM — first token, not last token
For voice, the LLM metric that matters is time to first token. Total completion time is nearly irrelevant, because we never wait for it.
What we do on the hot path:
- Stream every completion. Never request a blocking response.
- Keep the prompt lean. Long system prompts and bloated context inflate time to first token.
- Do retrieval fast. Business knowledge lives in pgvector; the RAG lookup runs before the model call and is tuned to stay cheap.
- Keep tool calls off the critical path where possible. A tool round-trip inserts a full extra model turn; when one is unavoidable, the agent acknowledges verbally first so the line is not silent.
Stage 4: speak the first sentence, not the reply
The output side mirrors the input side. We do not synthesize the full reply. As tokens stream in, we cut at the first clause boundary and hand that fragment to Aura-2 immediately. First audio reaches the caller while the model is still writing the rest of the answer.
// Flush to TTS at the first clause boundary, keep streaming the rest.
let buffer = "";
for await (const token of llmStream) {
buffer += token;
const cut = clauseBoundary(buffer); // . ? ! or , after enough chars
if (cut !== -1) {
tts.send(buffer.slice(0, cut + 1));
buffer = buffer.slice(cut + 1);
}
}
if (buffer.trim()) tts.send(buffer);
For TTS itself, first-byte latency matters far more than synthesis throughput. Aura-2 streams audio back fast enough that the clause-splitting above, not the synthesis, sets the pace.
Barge-in: latency's twin requirement
Callers interrupt. If your agent cannot be interrupted, low latency just means it starts talking over people sooner.
When VAD detects caller speech during playback, we stop TTS output, flush every queued audio buffer, and route the new speech into STT as a fresh turn. Flushing matters: if buffered audio keeps playing for two seconds after the caller starts talking, the agent feels deaf. The other half is echo discipline — the agent must not barge in on itself. WebRTC echo cancellation handles most of it, and we gate VAD sensitivity while the agent is speaking so residual echo does not trigger false interruptions.
The latency budget
These are budget allocations we engineer against, not lab benchmarks. What we actually measure end to end in production is roughly 0.7 s at the median.
| Stage | Budget | | --- | --- | | Turn detection (VAD + semantic endpointing) | ~250 ms | | STT finalization tail | ~50–100 ms | | LLM time to first token | ~200–300 ms | | TTS first audio byte | ~100–150 ms | | Transport and audio buffering | ~50–100 ms |
Two notes on measurement. First, measure voice-to-voice from the caller's perspective, not internal stage timings — buffering and transport hide between stages. Second, watch percentiles, not averages. LLM time to first token has real variance, and your p95 is what angry callers experience.
Geography is the silent line item. A cross-country round trip costs tens of milliseconds, and a naive deployment can stack several of them — caller to media server, media server to worker, worker to STT, worker to LLM. Keeping the worker and its upstream providers in nearby regions is free latency you only get by deciding to take it.
When not to chase this
Honesty about tradeoffs: aggressive endpointing is wrong for some callers. Elderly speakers and stressed callers pause longer mid-thought, and clipping them is worse than a slower agent. If your use case is reading long confirmations or disclosures, response latency barely matters. And if your call flow is a three-option menu, a plain IVR is cheaper and just as good. Sub-second response matters when the interaction is genuinely conversational — booking, qualifying, answering open questions.
Building this well is most of what our voice AI development practice does: the pipeline is only four boxes on a whiteboard, but every box hides a latency decision. If you are building a voice agent and fighting the same milliseconds, talk to us — we have already made most of these mistakes once, in production, so you don't have to.