Voice AI Agents in Production: What It Takes to Ship One People Do Not Hang Up On

The Short Answer

Shipping a voice AI agent is a turn-taking and repeatability problem, not a latency race. Humans reply to each other with a median gap near 100 ms, but squeezing an agent into a 300 ms budget roughly doubles how often it talks over the caller. Target 600 ms end to end, spend the savings on semantic endpointing, interruption recovery, and a clean audio path, and score reliability across consecutive runs, not one demo.

The 500 ms Target Is the Wrong Thing to Optimize

The standard advice is to pick the fastest speech-to-text, LLM, and text-to-speech you can afford, get under half a second, and ship. Component latency has commoditized, so that advice is now cheap to follow and differentiates nothing.

The human baseline is older than the current tooling. Stivers et al., writing in PNAS in 2009, measured 3,500 consecutive questions across 101 conversations in ten languages and found a cross-linguistic median response gap of about 100 ms. That is the bar a caller judges your agent against, not a vendor's marketing number.

You cannot hit it, and chasing it costs more than it buys. LiveKit's June 2026 benchmark of end-of-turn models put its own Turn Detector v1.0 at a 9.9 percent false-cutoff rate under a 300 ms latency budget, dropping to 4.5 percent at 600 ms. Hitting a 5 percent false-cutoff target cost 543 ms of mean latency; accepting 10 percent cost 295 ms. The aggressive target buys 250 ms of speed by doubling how often the agent interrupts the person paying for the call. Set the budget at 600 ms and stop.

Where the Latency Actually Lives

Most teams look for latency in model inference. It usually sits in the endpointing threshold. LiveKit's own guidance puts it plainly: a silence timeout of 800 ms adds nearly a full second to every response before the pipeline even starts.

That cost is additive on every turn. A faster LLM might save 80 ms. Moving from silence-based voice activity detection to semantic endpointing, where a model scores the probability that the speaker is finished rather than counting milliseconds of quiet, changes the shape of the system. The tooling is small and open: Pipecat's smart-turn-v3 is an 8M-parameter model running CPU inference in roughly 12 ms, and LiveKit's detector runs CPU-only in under 500 MB of RAM across 14 languages.

Check per-language numbers before assuming they generalize. LiveKit's model card reports a true negative rate ranging from 85.1 percent in Italian to 96.3 percent in Hindi, so mid-turn pause handling degrades unevenly by language. That behavior decides whether a caller reading a postcode off a screen gets cut off.

Published 2026 end-of-turn options:

Figures from LiveKit's June 2026 benchmark. Even the best number there means roughly one turn in twenty gets cut off, so build the recovery path first: detect the resumed turn, apologize once, and merge the fragments.

Short answers are the hard case. Daily.co reported Smart Turn v3.2 miscategorized single-word responses 40 percent less often after a training padding bug was fixed and short-utterance and cafe noise data added. An evaluation set full of complete sentences tests the easy half of the problem.

Barge-In Is Where the Transcript Starts Lying

A caller interrupts. The client stops playing audio. If nobody truncates the model's context, the history still holds eight seconds of speech the caller never heard, and the agent later references a confirmation number that was never spoken aloud.

Transport choice decides who fixes this. Per OpenAI's Realtime API conversations guide, the server tracks how much audio actually played and truncates automatically over WebRTC and SIP. Over WebSockets you send conversation.item.truncate yourself with the item id, content index, and audio end time, and skipping it leaves the desync in place for the rest of the call. OpenAI now terminates SIP directly, with an EU endpoint for data residency, so prefer SIP or WebRTC.

Speech-to-Speech or Cascaded: Choose by Artifact, Not by Speed

Speech-to-speech is faster and has fewer moving parts, which is why it is presented as the 2026 default. For most regulated commercial work it is the wrong choice, and the reason has nothing to do with quality.

Ask what artifact your business needs. If you require transcripts with timestamps for QA sampling, PII redaction, compliance review, or a warm handoff summary, a cascaded speech-to-text to LLM to text-to-speech pipeline produces that as a byproduct. Speech-to-speech makes you reconstruct it afterward, badly. Cost agrees: OpenAI's pricing puts gpt-realtime at $32.00 per 1M audio input tokens and $64.00 per 1M output, against $3.00 and $12.00 for Gemini 3.1 Flash Live.

What a Voice Agent Actually Costs Per Minute

Price the whole path, not the model. Published list prices, August 2026:

Two line items break naive forecasts.

Context growth is the first. OpenAI's realtime cost guide states that the entire conversation is sent to the model for each response, and bills user audio at one token per 100 ms and assistant audio at one token per 50 ms. A minute of caller speech is about 600 tokens and a minute of agent speech about 1,200, so minute twelve of a support call costs several times what minute two cost. Model your economics from a 60-second demo and a nine-minute average handle time will blow past the forecast. Enable prompt caching, set a retention ratio below 1.0 so truncation preserves the cache, and cap input with token limits.

Concurrency is the second. ElevenLabs caps concurrent calls at 10 on its $22 per month plan and 40 on its $990 plan, with burst at double rate. Retell charges $8 per concurrency past the first 20, Vapi $10 per line past 10. A campaign that fits your monthly minute budget can still fail at 9am on a Monday.

Compliance is a line item too. Vapi lists HIPAA at $2,000 per month and Zero Data Retention at $1,000, and a cascaded pipeline puts caller audio in front of four vendors with four retention defaults.

Your Demo Score Does Not Predict Production

This finding should change how you evaluate vendors. ServiceNow's EVA-Bench, published in May 2026, evaluated 12 voice agent systems spanning all three major architectures across 213 scenarios in three enterprise domains. No system exceeded 0.5 on both its accuracy and its experience metric at pass@1.

The reliability gap is the more useful number. The median difference between pass@k and pass^k on the accuracy metric was 0.44. Agents that can complete a task once frequently cannot complete it every time, and pass@1 measures the quantity that does not predict production behavior. EVA-Bench also measured mean performance deltas up to 0.314 under accent and noise perturbation, the drop your team never sees testing on its own voices.

The decision rule: if your evaluation is a happy-path script scored once, replace it. Score consecutive successes on the same task, and perturb the audio with accents and noise before believing any number a vendor or a pilot gives you.

The Audio Path Is the Product

The two most cited voice AI failures get read as evidence that the language problem is too hard. McDonald's ended its automated drive-through test with IBM in 2024, while saying voice ordering would still be part of its restaurants' future. Taco Bell rolled voice ordering to more than 500 drive-throughs, then told the Wall Street Journal it would coach franchisees to keep humans on the mic at peak hours, after clips circulated including a customer ordering 18,000 water cups to force a handoff.

A drive-through menu is a trivially constrained semantic space. What broke was acoustics, turn-taking, and the absence of an escape hatch, all engineering problems with known solutions.

Accuracy benchmarks will not warn you. A 2022 evaluation in Frontiers in Signal Processing found a recognizer trained on clean speech scoring 0.12 word error rate on clean audio and 0.79 on noise and network distorted speech. The top of the Artificial Analysis speech-to-text leaderboard clusters within tenths of a percent. Model choice is worth tenths, audio conditions tens.

Rules for the audio path:

• Put denoising upstream of the voice activity detector rather than raising VAD confidence or volume thresholds. That is Pipecat's guidance, and its defaults are start 0.2 s, stop 0.2 s, confidence 0.7, minimum volume 0.6.
• Re-benchmark speech-to-text on your own call recordings. Treat the leaderboard figure as a ceiling.
• Watch media-plane metrics, not container health. SIP signaling stays up while the RTP path dies, so the agent monologues into a dead leg while your health check reports green. Track packets per second, jitter, and a no-inbound-audio watchdog.
• Store raw audio alongside the transcript. Echo, double-talk, and codec artifacts are invisible in text.
• Provide a DTMF fallback for card numbers, confirmation codes, and postcodes. One wrong character invalidates the turn.
• Transfer to a human on the first clear request and log escalation rate as a product metric. The 18,000-water-cups clip existed because there was no obvious way out.
• Design for the tail. Deepgram publishes Flux end-of-turn latency at sub-300 ms median with a p95 of 1.5 seconds, and slow turns cluster on tool calls.

Treat Both the Voice and the Room as Untrusted

Voice is no longer usable as an identity factor. Pindrop's 2025 Voice Intelligence and Security Report, based on 1.2 billion customer calls, reported deepfake fraud attempts rising more than 1,300 percent in 2024, from roughly one a month to about seven a day, with retail contact centers seeing one attempt per 127 calls. If the voice sounding right feeds an authorization decision anywhere in your flow, that flow is compromised. Use an out-of-band factor for account recovery, password reset, and payments.

Ambient audio is an attack surface too. Research published in July 2026 on concurrent audio prompt injection found third-party audio played alongside a user's own speech hijacked multimodal voice agents at rates from 26.95 percent to 69.10 percent across 11 agents, with no access to model internals. On a speakerphone, kiosk, or conference bridge, gate every tool that moves money or reads back sensitive data behind a confirmation the model cannot issue to itself.

The same rule covers invented policy. Anything the agent can commit the company to, a refund, a price, a coverage rule, an appointment slot, goes through a validated tool call, never free-text generation. The recording is evidence of whatever the agent said.

Compliance Is a Design Constraint, Not a Legal Review

Outbound calls to US consumers are robocalls. The FCC ruled on 8 February 2024 that calls using AI-generated voices are artificial under the Telephone Consumer Protection Act, which means prior express consent, identification of the calling entity, and a working opt-out.

Any caller who may be in the EU must be told they are speaking with an AI. Article 50 transparency obligations of the EU AI Act became applicable on 2 August 2026, per the European Commission's FAQ. Disclosing at the top of the call costs two seconds and removes a category of risk.

How We Approach This at SASID

Our closest production experience here is call center QA, where we built automated coverage of 100 percent of calls for a client whose manual review had never reached 5 percent. The lesson that transferred to agents is that the audio, not the model, is where the surprises live. That work reflects how we deliver: proof of concept in days, production in four to eight weeks, and 90 days of monitoring, which for voice is when the real acoustic distribution shows up.

Get a Free Technical Assessment

If you are scoping or debugging a voice agent, we offer a free technical assessment: a 30-minute call about your use case, followed by a written roadmap within 48 hours covering architecture, turn-taking, evaluation design, cost per minute, and timeline. There is no obligation, and the roadmap is yours to keep. Book at sasid.ai.