An AI voice agent works by chaining four real-time systems — speech-to-text (STT), a large language model (LLM), text-to-speech (TTS), and a telephony layer — into a loop that hears a caller, decides what to say, and speaks back in under a second. The engineering challenge is not intelligence; it's latency. A natural human conversation has gaps of roughly 200 milliseconds between turns, so every millisecond the pipeline spends transcribing, reasoning, and synthesizing eats into the illusion of a real person. For businesses using voice agents on inbound leads, that speed compounds: leads contacted within five minutes are far more likely to qualify (MIT/Oldroyd Lead Response Management study), and an agent that answers in seconds captures revenue a human callback never reaches.
An AI voice agent is a real-time pipeline, not a single model
An AI voice agent is not one AI — it's an orchestrated pipeline of specialized models running concurrently, each optimized for one job. Understanding the components is the fastest way to understand why some agents sound robotic and others feel human.
The core stages, in order:
- Telephony / audio transport — carries the raw voice stream over VoIP (SIP, WebRTC) between the caller and the system.
- Speech-to-text (STT / ASR) — converts the incoming audio into text tokens in real time.
- Large language model (LLM) — interprets intent, tracks conversation state, and generates the reply.
- Text-to-speech (TTS) — synthesizes the reply text back into natural audio.
- Orchestration layer — manages turn-taking, interruptions, endpointing, and business logic (CRM lookups, transfers, scheduling).
Each stage adds latency, and latency is cumulative. If STT takes 300ms, the LLM takes 500ms, and TTS takes 300ms, you're already at 1.1 seconds before accounting for network transport. That's why leading systems stream every stage instead of waiting for one to finish before the next begins. The difference between a "chatbot on a phone" and an agent people mistake for a human is almost entirely in how tightly this pipeline is engineered.
Speech-to-text: turning sound waves into tokens in real time
Speech-to-text is the front door, and it has to be both fast and accurate while the caller is still talking. Modern voice agents use streaming ASR models that emit partial transcripts continuously rather than waiting for the speaker to finish a sentence.
Two technical problems dominate this stage:
- Endpointing — deciding when the caller has actually stopped speaking versus just pausing mid-thought. Cut them off too early and the agent interrupts; wait too long and the conversation feels sluggish. Good systems use a mix of silence-duration thresholds and semantic signals to predict turn-ends.
- Robustness — real phone calls have background noise, accents, cross-talk, and 8kHz narrowband audio that's far lower fidelity than a studio microphone.
Streaming STT models like OpenAI Whisper (in streaming configurations) and Deepgram's Nova family are common building blocks because they return interim results in tens of milliseconds. The best practice is to feed partial transcripts to the LLM early so it can begin "thinking" before the caller finishes — a technique sometimes called speculative processing. This overlap is one of the biggest levers for shaving total response time.
The LLM: intent, memory, and the reasoning layer
The LLM is the brain that decides what the agent says, but on a live call it must reason under a strict latency budget. This is where a voice agent diverges sharply from a text chatbot: nobody notices a two-second delay in a chat window, but two seconds of silence on a phone call feels like the line went dead.
To hit sub-second targets, engineers use several tactics:
- Streaming token generation — the LLM emits its answer word-by-word, and TTS starts speaking the first words before the full response is generated.
- Tight, structured prompts — a system prompt encodes the agent's role, guardrails, and available tools so it doesn't waste tokens reasoning about scope.
- Function calling / tool use — the model triggers actions mid-conversation: looking up a lead in the CRM, checking calendar availability, or qualifying a prospect against criteria.
- State tracking — the agent must remember what was said 30 seconds ago and stay consistent, which means feeding conversation history back into every turn.
Models from OpenAI (GPT-4 class), Anthropic (Claude), and Google (Gemini) are typical choices, often paired with smaller, faster models for simple routing decisions. The trade-off is constant: bigger models reason better but respond slower. Production voice agents frequently route easy turns to a fast model and reserve heavyweight reasoning for complex ones.
Text-to-speech: synthesizing a voice that doesn't sound synthetic
Text-to-speech is where the agent earns or loses the caller's trust, because humans detect artificial speech instantly. Modern neural TTS has closed most of the gap, producing natural prosody, breathing, and intonation — but it must do so while streaming audio out with minimal delay.
The key quality factors:
- Prosody and emphasis — placing stress on the right words so questions sound like questions and lists don't sound flat.
- Streaming synthesis — generating audio in chunks so the first syllable plays within ~200-300ms rather than after the whole sentence is rendered.
- Voice consistency — the same voice across an entire call, with no glitches on numbers, spelling, or unusual proper nouns.
Providers like ElevenLabs, Cartesia, and PlayHT specialize in low-latency streaming voices, and the field moves fast. The remaining tell-tale signs of AI are usually in edge cases: reading out a phone number digit by digit, handling an interruption gracefully, or expressing genuine-sounding empathy. Handling barge-in — when a caller talks over the agent — requires the TTS to stop instantly and the STT to re-engage, a coordination problem that lives in the orchestration layer, not TTS itself.
Orchestration and telephony: where most agents actually break
The orchestration layer is the hardest and least glamorous part of an AI voice agent, and it's where most implementations fall apart. It manages turn-taking, interruptions, silence, network jitter, and the business logic that makes the call useful instead of just conversational.
Responsibilities of the orchestration layer:
- Turn-taking logic — coordinating STT endpointing with LLM readiness so the agent replies at the right moment.
- Barge-in handling — killing playback the instant a human interrupts.
- Fallbacks — what happens when the LLM stalls, the caller says something off-script, or the network drops packets.
- Telephony integration — connecting to the phone network via providers like Twilio or Telnyx over SIP/WebRTC, handling call setup, DTMF (keypad) tones, and warm transfers to human reps.
- CRM and workflow actions — logging the call, storing the recording, transcript, and summary, and pushing qualified leads to the right person.
This is also where voice agents create measurable business value rather than novelty. When an inbound lead submits a form at 11pm, an orchestrated agent can call within seconds, qualify the prospect, and warm-transfer a hot buyer to a rep — capturing demand that would otherwise sit in a queue. That matters because 30-40% of inbound leads commonly arrive after hours, and roughly 78% of buyers purchase from the first company that responds (multiple sources). Platforms like Lead to Speed exist specifically to compress this loop, calling inbound leads in under 10 seconds and logging every recording, transcript, and AI summary automatically. For the full playbook on why response speed drives revenue, see the complete guide to speed to lead.
Latency budgeting: the 800-millisecond target
The single most important metric for an AI voice agent is round-trip latency — the time from when a caller stops speaking to when the agent starts speaking. The human conversational benchmark is roughly 200 milliseconds; anything under about 800ms feels natural, and past ~1.5 seconds callers start talking over the agent or assume it froze.
Here's how a well-engineered latency budget breaks down:
| Pipeline stage | Typical latency (streaming) | Optimization lever |
|---|---|---|
| Audio transport (network) | 50-150 ms | Regional edge servers, WebRTC |
| Speech-to-text (final) | 100-300 ms | Streaming ASR, early endpointing |
| LLM first token | 200-500 ms | Fast models, short prompts, streaming |
| Text-to-speech first audio | 100-300 ms | Streaming synthesis |
| Orchestration overhead | 50-150 ms | Co-located services, parallelism |
Latencies vary by provider, region, and load; treat these as illustrative ranges, not guarantees.
The trick is overlap. Because every stage streams, they run concurrently rather than in sequence — TTS begins speaking the first words while the LLM is still generating the rest, and STT feeds partial transcripts to the LLM before the caller finishes. That overlap is how systems that would naively total 1.5+ seconds deliver perceived response times under a second.
Where AI voice agents deliver real ROI — and where they don't
AI voice agents deliver the highest return on high-volume, time-sensitive, repeatable calls — and the lowest on complex, high-emotion conversations that demand human judgment. Knowing the difference is what separates a useful deployment from an expensive gimmick.
Strong fits today:
- Inbound lead qualification and instant callback — where speed is the entire value proposition. Contact within one minute drives dramatically higher conversion (Velocify research), and no human team is staffed to call every lead within seconds around the clock.
- Appointment reminders, confirmations, and rescheduling — structured, predictable, high-volume.
- After-hours coverage — capturing the 30-40% of leads that arrive when reps are offline.
- First-line triage — qualifying and routing before a human ever picks up.
Weaker fits:
- Complex negotiations, escalations, or emotionally charged support — where nuance and trust outweigh speed.
- Highly regulated conversations requiring strict compliance scripting and human sign-off.
The honest framing: AI voice agents don't replace great salespeople — they make sure a human ever gets the chance to talk to the lead. When average B2B lead response time runs ~29-47 hours depending on the study, an agent that responds in seconds isn't competing with your best rep; it's competing with a voicemail that never gets returned. For a deeper primer on the underlying concept, read what is speed to lead.
The build-vs-buy decision for voice agents
Building a production AI voice agent in-house is deceptively hard, because the demo is easy and the last 20% — latency, interruptions, edge cases, telephony reliability — is where months disappear. Most teams underestimate the orchestration layer and overestimate how much the underlying models matter.
A realistic comparison of the two paths:
| Factor | Build in-house | Buy a platform |
|---|---|---|
| Time to production | Months of engineering | Days to weeks |
| Latency tuning | You own the entire budget | Pre-optimized pipeline |
| Telephony reliability | Integrate + maintain yourself | Handled by vendor |
| CRM / transcript storage | Build it | Usually included |
| Model upgrades | Manual re-integration | Vendor-managed |
| Best for | Teams with deep ML + infra staff and unique needs | Sales/ops teams that need results fast |
Capabilities and pricing models (per-seat vs usage-based) differ widely between vendors and change frequently — verify current details directly before buying.
The strategic question isn't "can we build this?" — with today's APIs, most engineering teams can assemble a working prototype. The question is whether owning the maintenance of a real-time, always-on telephony system is a better use of your engineers than shipping your core product. For a sales or revenue team whose goal is contacting leads faster, a purpose-built platform that already solved latency, barge-in, warm transfers, and CRM logging will almost always reach production revenue faster than a from-scratch build.