Where the 900 Milliseconds Go in a Voice Agent

A voice agent that answers in 900 milliseconds feels distant. One that answers in 300 feels present. The gap between those two numbers is not one slow component; it is six components each contributing a defensible amount.

Here is the whole chain, in order, with the part that is usually mismeasured called out.

1. Endpointing

Before anything else can happen, the system has to decide the person stopped talking. This is the largest and least discussed item on the list.

Voice activity detection tells you there is silence. It cannot tell you whether that silence is the end of a turn or someone thinking about the second half of their sentence. So there is a wait threshold, and it is a straight trade: shorter means faster replies and more interruptions, longer means patience and lag.

Set it at 700ms and you have spent most of a conversational budget before a single byte reaches a model. Set it at 200ms and the agent talks over anyone who pauses mid-sentence, which people do constantly when reciting an order number.

The important thing is that this cost is invisible in most latency dashboards, because timing usually starts when transcription completes. The customer's clock started when they stopped speaking. If your measurement does not include endpointing, your reported latency is missing its biggest term.

2. Transcription

Streaming transcribers emit partial results continuously, so by the time endpointing fires, most of the utterance is already transcribed. The remaining cost is finalisation, not transcription.

Batch transcription, where audio is sent after the turn ends, is where this stage gets expensive. The work is identical; it is simply all scheduled after the customer has begun waiting rather than while they were talking.

3. The model

The number that matters here is time to first token, not total generation time.

Once the first token exists, speech synthesis can start, and once speech starts the customer is no longer in silence. Everything after that overlaps with audio the customer is listening to. A model that produces its first token in 200ms and finishes in 2 seconds beats a model that produces its first token in 600ms and finishes in 1 second, comfortably, despite being slower by the metric most benchmarks report.

This is also where prompt size shows up. A long system prompt and a long history are paid at the start of every turn. Prompt caching helps a great deal here and is worth doing for exactly this reason.

4. Speech synthesis

Same principle. Time to first audio chunk is the number; total synthesis time is not.

A streaming voice starts speaking on the first sentence while the rest is still being generated. A non-streaming one waits for the complete text, synthesises it, and then plays it, which serialises two stages that could have overlapped and adds their full durations together.

5. Telephony

Carrier and network transit. Typically 50 to 150ms, largely outside your control, and worth knowing rather than fighting.

What is inside your control is where your infrastructure sits relative to the phone number. An agent served from a different continent to its caller pays the round trip on every leg of the chain, several times per turn.

6. Your own tools

If the turn involves a lookup, that lookup is on the critical path and it is usually the least optimised thing in the whole system, because it was written for a web request where 800ms was fine.

It is not fine here. An API that is comfortably fast for a page load can single-handedly break a conversation.

Adding up, and where to actually cut

Realistic pipeline, nothing pathological:

endpointing      500ms
transcription     50ms   (streaming)
model TTFT       350ms
TTS first audio  200ms
network          100ms
-----------------------
                1200ms to first sound

Every stage is reasonable. The total is bad.

Notice where the money is. Endpointing is the largest line and gets the least attention, because it feels like a configuration value rather than engineering. Dropping it from 500ms to 250ms and handling the resulting interruptions properly buys more than switching to a faster model.

The second biggest win is not a stage at all. It is making sure every stage that can stream, streams. Endpointing, transcription, generation, and synthesis are all capable of overlapping. Run them in sequence and you pay the sum. Overlap them and you pay something close to the maximum.

The measurement that keeps you honest

Instrument one number and put it on a wall: customer stops speaking, to customer hears sound.

Not model latency. Not p50 API response. That single interval is the only one the person on the phone experiences, it includes every stage including the ones nobody owns, and it is the number that decides whether they describe your agent as sharp or as strange.

Most teams measure the middle of the chain, optimise the middle of the chain, and cannot work out why the calls still feel slow.