Voice AI Agent Architecture in 2026:Latency, Barge-In, and Telephony

A text-based chat agent has no deadline. A user typing into a chat window will wait two, three, even five extra seconds for a genuinely good answer without registering it as a problem — the interaction is asynchronous by nature, and a thinking indicator absorbs the wait. A voice agent has no such grace period, because it is bound by a timing expectation nobody had to teach anyone: the rhythm of human conversation itself.

That rhythm is not a vague impression — it has been measured directly. A landmark cross-linguistic study published in the Proceedings of the National Academy of Sciences in 2009 recorded natural conversation across ten typologically unrelated languages, from English and Japanese to Tzeltal and Yélî Dnye, and found a strikingly consistent pattern: the gap between one speaker finishing a turn and the next speaker starting has an overall mode of 0 milliseconds and a cross-linguistic median of about 100 milliseconds, with the large majority of turn transitions landing between 0 and 200 milliseconds. That number is close to the physical limit of how quickly a human can plan and launch a spoken response — people are, in effect, starting to formulate their reply before the other person has finished talking.

This guide is written for the operator deciding how to architect a voice agent that will actually run on real phone lines — not for a speech researcher already deep in acoustic modeling. It covers where the latency budget actually goes, how to choose between a cascade pipeline and a unified speech-to-speech model, how barge-in and turn-taking are engineered in practice, what telephony integration actually requires, and — because vendor pages routinely skip this — the regulatory and security questions that decide whether a voice agent should be making outbound calls at all.

"Fast model" and "fast agent" are not the same claim, and conflating them is the most common mistake in this space. A voice agent's end-to-end latency — measured from the instant a caller stops talking to the instant the agent's audio starts playing — is the sum of several independent stages, each with its own budget and its own levers.

Two engineering patterns do more to close this gap than swapping any single vendor for a marginally faster one. The first is streaming rather than waiting at every stage: streaming partial transcripts from STT into the LLM before the caller has finished their sentence, and streaming synthesized audio from TTS as soon as the first few words of a response are ready rather than waiting for the full utterance to render. The second is model-size routing — using a smaller, cheaper, faster model for short, predictable turns (confirmations, simple lookups) and reserving a larger model only for turns that genuinely need it, rather than paying the same latency tax on every single exchange in a call.

Our own general-purpose guide to AI agent latency engineering covers streaming, parallel tool calls, speculative decoding, and model routing in more depth for agents generally; everything in that guide applies to the LLM stage of a voice pipeline specifically, layered underneath the STT and TTS latency this article covers on top of it.

There are two fundamentally different ways to architect the core of a voice agent, and the choice shapes almost every other decision downstream of it.

A cascade pipeline keeps speech-to-text, the language model, and text-to-speech as separate, swappable components — Deepgram or AssemblyAI for transcription, any LLM provider, and ElevenLabs or Cartesia for synthesis, stitched together with a framework like Pipecat or LiveKit Agents. Its advantage is visibility and flexibility: you have a full text transcript at every stage for logging and evaluation, and you can swap any single component — a better TTS voice, a cheaper STT provider — without touching the rest of the stack. Its cost is stitching latency: every hop between vendors adds a network round trip on top of each stage's own processing time.

A unified speech-to-speech model collapses that into one model that takes audio in and produces audio out directly, with no intermediate text-generation stage a developer can inspect. OpenAI's Realtime API (the gpt-realtime model family), Google's Gemini Live API — generally available on Vertex AI and built on Gemini's Flash Native Audio model — and Amazon's Nova 2 Sonic on Bedrock are the three major production examples as of 2026. Nova 2 Sonic in particular now integrates natively with Amazon Connect and with third-party telephony providers including Twilio, Vonage, and AudioCodes, as well as the open-source frameworks Pipecat and LiveKit, which narrows one of the traditional gaps between the unified and cascade approaches.

Barge-in — a caller interrupting the agent mid-sentence and the agent actually stopping to listen — is the single most under-tested mechanism in voice agent builds, and it's the one most likely to make a technically fast agent still feel broken to a real caller.

The mechanism underneath barge-in is Voice Activity Detection (VAD), scoring the incoming audio stream continuously while the agent is still speaking. A production-grade implementation combines three signals rather than relying on any one alone: an energy threshold (commonly tuned in the range of -45 to -35 dBFS) to catch genuine speech volume, a voice-versus-noise classifier to distinguish an actual human voice from background sound, and a minimum-duration guard — typically 200 to 300 milliseconds of sustained voice — before the system commits to treating it as a real interruption rather than a cough, a door closing, or a moment of line noise.

• Too sensitive: The agent stops talking every time a dog barks, a TV plays in the background, or the caller clears their throat — it feels twitchy and unreliable rather than responsive.
• Too conservative: Callers learn they have to raise their voice or talk over the agent repeatedly to be heard at all, which reads as the agent not listening rather than as the agent being deliberate.
• Correctly tuned: The agent yields the floor within roughly 200-300ms of genuine speech onset and stays quiet through background noise — which requires tuning against real phone-line audio, not clean studio recordings, before launch.

Turn-taking is the conversational policy layered on top of barge-in detection: deciding not just whether the caller is speaking, but whether a pause means they're finished or just thinking. Semantic turn-detection — using the content of what's been said so far, not just silence duration, to judge whether a turn is actually complete — is where the newer unified speech-to-speech models and modern VAD libraries are investing specifically, since a caller who pauses mid-sentence to think should not trigger the agent jumping in.

Everything above assumes audio is already flowing between a caller and your agent — telephony is the layer that actually makes that connection exist, and it's where a voice agent build most often collides with infrastructure most AI teams have never had to touch before.

A call from a traditional phone number travels over the PSTN (the Public Switched Telephone Network) and needs to be bridged into an IP-based system your agent can process — that bridge is handled by SIP (Session Initiation Protocol) trunking, the standard signaling protocol for setting up, managing, and tearing down a call over an IP network. A web-based voice interaction — a caller talking to an agent embedded in a browser or app rather than dialing a phone number — instead typically uses WebRTC, the browser-native standard for real-time audio and video. A production voice agent handling real phone calls needs to speak SIP at minimum, and increasingly WebRTC as well if it also supports in-app or in-browser voice interactions.

Three tiers of provider sit underneath this, and picking the right one depends on how much of the stack you want to own. A CPaaS provider that owns its own telecom infrastructure — Telnyx or Bandwidth — gives you the lowest-level control and typically the lowest per-minute cost at scale. A CPaaS provider that orchestrates across partner carriers rather than owning the network — Twilio, historically the default choice — trades some of that control for broader ecosystem maturity and documentation. And an open-source real-time media layer — LiveKit's WebRTC-based media server, paired with LiveKit Agents, or Pipecat, the BSD-licensed framework built by Daily for pipelining audio through STT, an LLM, and TTS — sits a level above raw telephony, letting you assemble your own pipeline against whichever underlying SIP or WebRTC provider you choose.

Voice AI infrastructure attracted serious capital through 2026, and the funding activity is a genuinely useful signal for which platforms are likely to still be well-supported a year from now — not a substitute for evaluating the product directly.

A methodology note, because this table is doing comparative work: we scored these vendors here only on funding status, scale signal (reported call volume, named enterprise customers), and pricing where publicly stated — all dated and attributed to the specific source. We deliberately did not score, rank, or repeat any vendor's own claimed accuracy, containment, deflection, or ROI figures, because none of the vendors above published a methodology alongside those numbers that would let a reader verify them independently. If a vendor's sales material cites a specific accuracy or resolution-rate percentage, ask for the underlying evaluation methodology before treating it as comparable to another vendor's number — the two are very unlikely to have been measured the same way.

For a deeper comparison of the orchestration frameworks underneath several of these platforms — not voice-specific, but directly relevant if you're routing tool calls or multi-agent handoffs behind a voice front end — see our AI agent orchestration frameworks guide.

Voice agents that call real phone numbers — especially outbound — sit inside one of the most actively litigated corners of telecom law in the country, and getting the precise status of each rule right matters more here than almost anywhere else in this guide, because "in force," "vacated," and "proposed" carry genuinely different obligations.

The core substantive rule is settled and in force: the FCC's Declaratory Ruling FCC 24-17, adopted February 8, 2024, affirmed that an AI-generated voice qualifies as an "artificial voice" under the Telephone Consumer Protection Act — meaning a robocall using an AI-generated voice is subject to the same prior-consent requirements as any prerecorded call, and we found no indication this specific ruling has been vacated or stayed.

What is genuinely unsettled is the mechanics of that consent. The Eleventh Circuit vacated the FCC's stricter "one-to-one consent" rule nationwide on January 24, 2025, in Insurance Marketing Coalition Limited v. FCC, holding the FCC had exceeded its statutory authority; the FCC subsequently reinstated the prior consent rule. Just over a year later, on February 26, 2026, the Fifth Circuit held in Bradford v. Sovereign Pest Control of TX, Inc. that the TCPA permits either oral or written consent for autodialed or artificial-voice telemarketing calls, explicitly rejecting the FCC's written-consent-only reading — a holding that applies within that circuit specifically, per the Supreme Court's Loper Bright framework giving courts latitude to interpret the statute independent of FCC regulations, rather than a nationwide rule change. The practical upshot: verify your specific consent posture against current law in the jurisdictions you actually operate in, rather than assuming a single compliance template holds everywhere.

A separate, still-proposed rule would require disclosing that a caller is talking to AI at all. The FCC adopted a Notice of Proposed Rulemaking on August 7, 2024 that would establish new consent and identification disclosure requirements specifically for AI-generated calls and texts. It has not been finalized into a binding rule as of this writing, and industry trackers following the docket estimate a final rule is unlikely before late 2026 at the earliest, with current FCC leadership's stated deregulatory priorities a real factor in further delay. Treat it as a rule to design toward, not one you're already bound by federally — while checking independently whether your state has moved faster with its own disclosure requirement.

Finally, every call your agent places or receives over the PSTN runs through the STIR/SHAKEN caller ID authentication framework, which your telephony or CPaaS provider is required to implement on the IP portions of its network, alongside an annually recertified robocall mitigation plan filed in the FCC's Robocall Mitigation Database. New third-party signing rules effective September 2025 require a provider with a signing obligation to sign calls with its own certificate rather than a third party's — worth confirming directly with your telephony vendor if you're placing outbound calls at meaningful volume.

A voice agent that listens to an open phone line is processing untrusted input continuously, in real time, from anyone who can get a call connected to it — and prompt injection through that audio channel is not a solved problem. Treat it as a genuine, live attack surface, not a hypothetical one this guide is raising out of caution.

Audio is a structurally different injection surface than text or an uploaded image, because it doesn't require a file upload or a crafted document — it's captured continuously over an open line, which makes an always-listening agent exposed to instructions spoken directly into the call in real time. Published research through 2026 has demonstrated "concurrent" audio injection techniques that embed semantically instruction-like content designed to hijack a multimodal agent's attention away from a legitimate speaker's concurrent speech, and separate benchmarking work has specifically measured how large audio-language models respond when adversarial audio is injected alongside normal conversational input.

Voice cloning adds a second, related risk specific to this modality: a caller impersonating another person's voice to social-engineer either a human agent or an AI voice agent into an action it shouldn't take. Caller-ID spoofing compounds it further — the same STIR/SHAKEN framework covered above exists specifically because caller ID has historically been trivial to fake.

Voice agent economics are genuinely per-minute, not per-request the way most text agent pricing works, and the number varies enough by architecture that a vendor's headline price is rarely the whole answer.

On the unified speech-to-speech side, OpenAI's Realtime API prices audio input at $32 per million tokens — roughly $0.06 per minute of audio — and audio output at $64 per million tokens, roughly $0.24 per minute, with a substantially cheaper cached-input rate available for repeated system context across a call. On the cascade side, ElevenLabs cut its conversational AI pricing to $0.10 per minute in early 2026, which covers its own text-to-speech and orchestration layer but not a separately billed STT or LLM cost if you're not running its full bundled stack end to end. Telephony itself — the SIP trunk or CPaaS leg carrying the actual phone call — is typically a smaller, separate per-minute line item on top of whichever AI stack sits behind it.

The honest way to budget this is to model your full stack's blended per-minute cost against your real expected call volume and average call length, not to compare a single vendor's headline number against another's — a bundled platform's all-in per-minute price and a self-assembled cascade stack's summed STT-plus-LLM-plus-TTS-plus-telephony cost are answering the same question from two different starting points.

To make the latency and cost tradeoffs concrete, consider an illustrative scenario — not a client engagement, but a worked example built from the real figures cited throughout this guide. A regional home-services operator running 40 locations fields roughly 3,000 inbound dispatch calls a month, each averaging four minutes: routing a caller to the right local crew, confirming an appointment window, and escalating anything unusual to a human dispatcher.

At 3,000 calls averaging four minutes, that's 12,000 minutes of monthly call volume. A cascade pipeline built on a mid-tier STT provider, a routing-sized LLM for the simple confirmation turns, and a streaming TTS voice could reasonably target the 600ms-1.7s latency band this guide describes as achievable, with per-minute AI costs — excluding telephony — landing somewhere between the ElevenLabs-style $0.10/minute conversational tier and a fuller custom stack, depending on which STT and LLM combination is chosen; telephony itself adds a separate, smaller per-minute charge from whichever SIP or CPaaS provider carries the calls. The arithmetic that actually matters for a decision here is comparative, not a single number: at this volume, the operator is weighing a bundled platform's simpler all-in per-minute price against the lower marginal cost, but higher up-front engineering cost, of a custom cascade stack — and the honest answer depends on the operator's specific call complexity and growth trajectory, not on which architecture is abstractly "better."

The part of this scenario most operators underestimate isn't the AI cost at all — it's the barge-in tuning work. A dispatch line fields calls from job sites, moving trucks, and outdoor locations with real background noise, which is exactly the condition under which a poorly tuned VAD either interrupts constantly or can't be interrupted at all. Budgeting real time against that specific tuning step, against real call recordings rather than clean studio audio, is what separates a pilot that works in a demo from one that holds up on a Tuesday afternoon with a crew running a leaf blower in the background.

Generalizing from the tradeoffs covered above into an actual build sequence for an operator adding a voice agent:

The order matters for the same reason it does in any agent build: the expensive mistakes in voice agent projects are almost always architectural decisions made too early or too late, not model-quality problems. Deciding cascade versus unified before writing integration code, and deciding whether to own telephony infrastructure only after hitting a specific limitation on a bundled platform, are the two decisions most worth getting right in the order shown above rather than reversing.

Voice agent work scopes the same way the rest of our agent-architecture work does: a discovery phase that maps your real latency and call-volume requirements before any code changes, a single-workflow build for a first, well-scoped call type, a platform build for operators handling several call types, and an enterprise band for organizations with real regulatory exposure.

Senior-led delivery runs $150 to $225 per hour, retainers run $2,500 to $9,500 per month, and every engagement carries a 30-day post-launch warranty. Book a discovery call at calendly.com/frenchydigital/discovery-call or call +1 (424) 272-5601, and you receive a written, fixed-price phased proposal within five business days.

A short, consolidated list worth checking against any voice agent vendor or platform pitch, gathered from the specific claims this guide had to verify or refuse above.

This guide is explicit about where its own verification stopped. The specific latency breakdown in the latency-budget table (100-300ms STT, 350ms-1,000ms LLM, 90-200ms TTS, 50-200ms network) reflects a range synthesized from vendor-published technical guides and benchmark pages, attributed as such — we did not independently run a controlled latency benchmark across these vendors ourselves, and a specific deployment's real numbers will vary by provider, region, and call complexity. Cartesia's reported synthesis-latency figures and other vendor-specific performance claims referenced in this space generally are vendor-reported rather than independently audited, and we have not included any specific unattributed number as fact.

We did not independently verify Retell AI's reported $50M ARR figure or Bland AI's outbound-volume positioning beyond the third-party industry coverage cited — both are reported here as industry-reported claims, not audited figures. We also did not test any vendor's platform hands-on as part of writing this guide; the architecture, latency, and telephony descriptions above reflect each vendor's own documentation and named press coverage, not our own benchmarking.

The regulatory landscape described in this guide — particularly the Fifth Circuit's February 2026 ruling and the still-pending FCC disclosure NPRM — is genuinely unsettled and actively moving; a reader making a real compliance decision should verify current status directly with counsel before it factors into a launch timeline, rather than treating this guide's September 2026 snapshot as still current by the time they read it. Finally, this is a fast-moving infrastructure space — several of the vendors and models referenced shipped meaningful updates within the twelve months before this guide was written, and per-minute pricing in particular should be reconfirmed directly against each vendor's current pricing page before it factors into a budget.