How to Build an AI Voice Agent: Architecture, APIs, and What Production Systems Actually Require

A production AI voice agent requires four systems working in sequence under 800 milliseconds of total latency: speech-to-text (STT), a language model for intent and response generation, text-to-speech (TTS), and telephony integration for call routing and state management. Most tutorials stop at the first two. The engineering that separates a demo from a system that answers 500 calls a day lives in the last two, plus the orchestration layer that keeps all four synchronized during a live conversation.

What does a production AI voice agent architecture look like?

A production voice agent is a real-time pipeline, not a chatbot with a microphone attached. The caller speaks, the STT engine transcribes in streaming mode (not waiting for silence), the LLM generates a response based on conversation context and business rules, the TTS engine converts that response to natural speech, and the telephony layer delivers it back to the caller while managing call state, transfers, and recordings.

The four-stage pipeline looks simple on a whiteboard. In production, every stage introduces latency, and latency is the difference between a conversation that feels natural and one where the caller hangs up. Human conversation tolerates about 300 to 500 milliseconds of pause before it feels unnatural. Your entire pipeline, from the moment the caller stops speaking to the moment they hear the first syllable of the response, needs to fit inside that window.

The fifth component, often missing from architecture diagrams, is the orchestration layer. This manages conversation state (what has been said, what the caller wants, where they are in a workflow), decides when to interrupt the caller vs. wait for them to finish, handles barge-in (the caller speaking over the agent), and coordinates with external systems like CRMs, scheduling tools, and knowledge bases.

Which speech-to-text APIs work for real-time voice agents?

Streaming STT is non-negotiable. Batch transcription (upload audio, wait for text) adds 2 to 5 seconds of latency, which kills any conversational experience. The STT engine must accept audio chunks in real time and return partial transcripts as the caller speaks.

Deepgram is the current production default for most custom voice agents. It offers streaming transcription with 100 to 300 millisecond latency, handles background noise and accents well, and provides endpointing (detecting when the caller has finished a thought). Google Cloud Speech-to-Text v2 is a strong alternative with better multilingual support but slightly higher latency. OpenAI Whisper is excellent for accuracy but was designed for batch processing; the streaming variants available through third-party wrappers add latency that makes them less suitable for sub-second pipelines.

The critical STT decision is endpointing sensitivity. Set it too aggressive and the engine cuts off callers mid-sentence. Set it too loose and the agent waits 2 seconds after every utterance before responding. Production systems tune this per use case: appointment booking tolerates tighter endpointing because utterances are short ("Tuesday at 3"), while complaint handling needs looser settings because callers speak in longer, less predictable patterns.

How does the language model handle live conversation context?

The LLM in a voice agent does not receive a single prompt and return a single response. It maintains a rolling conversation transcript, enriched with system instructions that define the agent's persona, business rules, and available actions. Every caller utterance appends to this transcript, and the LLM generates the next response based on the full conversation history plus any context retrieved from external systems.

Claude and GPT-4o are the two models most commonly used in production voice agents as of 2026. Claude handles longer system prompts with more consistent instruction-following, which matters when the agent needs to enforce business rules ("never quote a price over the phone," "always confirm the caller's identity before accessing their account"). GPT-4o has a slight edge in raw speed for shorter responses. Both support function calling, which is how the agent triggers actions: booking an appointment, looking up an order, transferring to a human.

Streaming the LLM response is essential. Instead of waiting for the complete response, the TTS engine starts converting the first sentence to speech while the LLM is still generating the second sentence. This shaves 500 to 1,500 milliseconds off perceived latency. The orchestration layer handles the complexity: if the LLM's first sentence is "Let me check that for you," the TTS starts speaking immediately while the LLM runs a function call to retrieve data from a CRM or scheduling system.

Which text-to-speech engines sound natural enough for phone calls?

TTS quality has improved dramatically since 2024, but the gap between "sounds good in a demo" and "sounds good on a phone call" is still significant. Phone audio is compressed, bandwidth-limited, and often played through a speaker on the caller's end. A voice that sounds human in a headphone demo can sound robotic through a cell phone speaker.

ElevenLabs is the current quality leader for natural-sounding TTS with streaming support and low latency (under 200 milliseconds to first audio byte). Their voice cloning allows businesses to create a consistent brand voice. PlayHT and Cartesia are strong alternatives with competitive pricing for high-volume use. OpenAI's TTS API produces natural output but has higher latency than ElevenLabs for streaming use cases.

The production decision most teams miss is audio format optimization. TTS engines default to high-quality formats (24kHz+ PCM or Opus) that sound excellent over WebRTC but degrade when transcoded to the 8kHz G.711 mu-law format that traditional phone networks use. Building for telephony means testing the voice output through actual phone lines, not browser previews. A voice that sounds warm in a browser demo can sound tinny and compressed through a PSTN call.

How does the telephony layer connect the agent to real phone calls?

The telephony layer is where most voice agent projects stall. Getting an LLM to generate good responses and a TTS engine to read them aloud is a weekend project. Connecting that pipeline to actual phone numbers, handling call transfers, managing voicemail, recording calls for compliance, and scaling to hundreds of concurrent calls is a production engineering challenge.

Twilio is the default telephony provider for custom voice agents. Their Media Streams API provides a WebSocket connection that delivers raw call audio in real time, which your pipeline processes through STT, LLM, and TTS before sending the response audio back through the same WebSocket. Twilio handles the phone number provisioning, call routing, recording, and PSTN connectivity. Vonage and Telnyx offer similar capabilities with different pricing models: Telnyx is typically cheaper for high-volume use cases, while Twilio has the largest ecosystem of integrations.

Call transfer is the feature that separates a voice agent from a voice menu. When the agent determines that a caller needs a human (complex complaint, high-value opportunity, regulatory requirement), it must transfer the call with context. The human agent receiving the transfer should see a summary of the conversation so far, the caller's identified intent, and any data retrieved during the automated portion. A cold transfer ("please hold while I transfer you") followed by the caller repeating everything to a human destroys the value proposition of the voice agent.

What is the hardest engineering problem in production voice agents?

Barge-in handling. This is when the caller starts speaking while the agent is still talking. In human conversation, this happens constantly: people interrupt, talk over each other, and both parties adjust. In a voice agent, barge-in creates a cascade of problems. The agent's own TTS output feeds back into the STT engine, creating a loop where the agent hears itself. The caller's new utterance overlaps with the agent's ongoing response. The LLM needs to decide whether the new utterance is an interruption ("actually, wait") or an acknowledgment ("yeah, uh-huh") that doesn't require stopping.

Production systems handle barge-in with echo cancellation (removing the agent's own voice from the incoming audio stream), voice activity detection (distinguishing caller speech from background noise during agent playback), and an interrupt threshold (how much caller speech triggers the agent to stop talking and listen). Getting this wrong is the single most common reason voice agents feel robotic: either the agent never stops talking when interrupted, or it stops at every background noise.

The second hardest problem is silence handling. When a caller pauses for 3 seconds, is the pause intentional (they're thinking), accidental (they're distracted), or a signal that they're waiting for the agent to speak? The agent needs different responses for each: wait patiently, prompt gently ("are you still there?"), or continue with the next step. Production systems use a tiered timeout: 2 seconds triggers nothing, 4 seconds triggers a gentle prompt, 8 seconds triggers a check-in, 15 seconds triggers a graceful call ending.

What does the orchestration layer actually manage?

The orchestration layer is the custom code that sits between all four pipeline stages and manages the conversation as a stateful workflow. It is not a library or an API. It is the application logic specific to what the voice agent does for the business.

For an appointment-booking voice agent, the orchestration layer manages: caller identification (matching the phone number to an existing customer record or collecting new caller information), intent classification (new appointment, reschedule, cancellation, general question), availability checking (querying the scheduling system in real time), slot confirmation (repeating the proposed time and getting verbal confirmation), confirmation delivery (sending an SMS with the appointment details after the call), and escalation rules (when to transfer to a human).

This layer also manages the LLM's system prompt dynamically. When the agent identifies that the caller is an existing high-value customer, it can inject that context into the system prompt so the LLM adjusts its tone and priorities. When the conversation enters a compliance-sensitive flow (collecting payment information, confirming medical details), the orchestration layer swaps to a stricter prompt that prevents the LLM from improvising.

How do you handle latency across the full pipeline?

Total pipeline latency breaks down roughly as follows in a well-optimized system: STT streaming adds 100 to 300 milliseconds, LLM inference adds 200 to 800 milliseconds (depending on response length and model), TTS first-byte latency adds 100 to 200 milliseconds, and telephony round-trip adds 50 to 100 milliseconds. Unoptimized, this totals 1 to 2 seconds. Optimized with streaming at every stage, it drops to 400 to 700 milliseconds.

The optimization that makes the biggest difference is sentence-level streaming. Instead of waiting for the LLM to generate its complete response, the orchestration layer detects sentence boundaries in the LLM's streaming output and sends each completed sentence to TTS immediately. The caller hears the first sentence while the LLM is still generating the second. This technique alone can reduce perceived latency by 40 to 60 percent for multi-sentence responses.

Filler phrases are the other production trick. When the orchestration layer detects that a function call (checking a database, querying an API) will take more than 500 milliseconds, it injects a natural filler: "Let me pull that up for you" or "One moment while I check." The TTS speaks the filler while the function call executes in parallel. Without fillers, the caller hears dead silence during every data lookup, which feels broken.

What are the common voice agent platforms and when should you build custom?

Vapi, Retell AI, and Bland AI are the three most common voice agent platforms as of 2026. Each provides a managed pipeline (STT + LLM + TTS + telephony) with a configuration layer that handles most of the orchestration. For standard use cases like appointment booking, lead qualification, and FAQ handling, these platforms reduce development time from 8 to 12 weeks to 1 to 2 weeks.

The platforms break when the use case requires any of: custom barge-in logic (industry-specific interrupt handling), multi-system orchestration during the call (checking three different backends before responding), complex conditional workflows (different conversation paths based on real-time data from external systems), compliance recording with specific redaction requirements, or call volumes above 200 concurrent calls with custom SLAs. At that point, the platform's configuration layer becomes a constraint, and a custom build on top of the raw APIs (Deepgram + Claude/GPT-4o + ElevenLabs + Twilio) gives the engineering team full control over every stage.

The cost trade-off is straightforward. Platforms charge per minute of call time, typically $0.07 to $0.15 per minute all-in. A custom build using direct APIs costs $0.03 to $0.06 per minute at scale but requires $40,000 to $80,000 in development and $2,000 to $5,000 per month in infrastructure and monitoring. The breakeven is usually around 20,000 to 30,000 minutes per month: below that, platforms are cheaper; above that, custom is cheaper and gives more control.

What does it cost to build and run a custom AI voice agent?

A production voice agent for a single use case (answering calls, booking appointments, routing inquiries for one business) costs $40,000 to $70,000 to build over 10 to 14 weeks. A multi-use-case system handling inbound and outbound calls with integrations to CRM, scheduling, and billing systems runs $80,000 to $150,000 over 16 to 24 weeks. Operating costs (API usage, telephony, monitoring, model inference) run $2,000 to $8,000 per month depending on call volume.

The largest variable cost is LLM inference. A 2-minute call generates roughly 1,500 to 2,500 tokens of LLM input/output. At Claude or GPT-4o pricing, that is $0.01 to $0.03 per call for the LLM alone. STT adds $0.005 to $0.01 per minute. TTS adds $0.01 to $0.03 per minute. Telephony adds $0.01 to $0.02 per minute. Total per-minute cost for a custom-built system: $0.03 to $0.06. At 10,000 calls per month averaging 3 minutes each, that is $900 to $1,800 per month in variable API costs.

Compare this to human agents. A full-time receptionist in the US costs $35,000 to $50,000 per year. An answering service charges $1.50 to $3.00 per call. A voice agent handling 10,000 calls per month at $0.15 per minute (platform pricing, 3-minute average) costs $4,500 per month, or $54,000 per year. The same volume on a custom build costs $1,800 per month variable plus amortized development cost. The ROI math works for any business handling more than 2,000 calls per month with predictable, structured conversations.

What separates a voice agent demo from a production system?

Demos handle the happy path: the caller speaks clearly, asks a straightforward question, and the agent responds correctly. Production handles everything else. The caller has a thick accent. The caller puts the phone on speaker in a noisy car. The caller asks something outside the agent's domain. The caller gets angry. The call drops and the caller calls back expecting the agent to remember the previous conversation. The scheduling system returns an error. The LLM hallucinates a time slot that does not exist.

Production voice agents need: graceful fallback ("I'm having trouble understanding. Let me transfer you to someone who can help."), session persistence (caller ID lookup so returning callers don't repeat themselves), monitoring and alerting (dashboards showing call completion rate, average handle time, transfer rate, caller satisfaction), call recording with transcription for quality review, and a feedback loop where failed calls are reviewed, patterns identified, and the system prompt or orchestration logic updated weekly.

Madgeek built a call quality monitoring system for a BPO operation that scaled from 50 to 80+ agents in 3 months. The production lesson from that project: voice systems fail silently. A web application shows an error page when something breaks. A voice agent just sounds confused, and the caller hangs up without telling anyone. The monitoring layer that detects these silent failures (calls shorter than 30 seconds, calls with more than 3 consecutive misunderstandings, calls where the agent repeats itself) is what makes the system improvable over time.

What should you evaluate before starting a voice agent project?

Start with your call volume and call type distribution. Pull 30 days of call records and categorize them: how many calls per day, what percentage are simple (appointment booking, order status, business hours), what percentage are complex (complaints, multi-step troubleshooting, sensitive conversations). Voice agents deliver ROI fastest on the simple calls. If 60% or more of your calls fall into 3 to 5 predictable categories, a voice agent handles those while your human team focuses on the complex 40%.

Use a platform (Vapi, Retell AI, Bland AI) if your use case is standard appointment booking, lead qualification, or FAQ handling with fewer than 20,000 minutes per month. Use those platforms for a proof of concept even if you plan to build custom later: a $500 platform prototype in 1 week tells you more about caller behavior than a $40,000 custom build in 12 weeks.

Build custom when the platform's configuration layer limits your workflow logic, when you need integrations with more than two external systems during a live call, when compliance requirements demand specific recording, redaction, or audit capabilities, or when your call volume makes the per-minute platform pricing more expensive than owning the infrastructure. Madgeek scopes custom voice agent builds as fixed-price engagements: a 2-week discovery sprint defines the conversation flows, integration requirements, and latency targets before any code is written.