How to Build a Voice AI Receptionist That Answers Calls Well
A voice AI receptionist is a phone or web voice agent that answers calls, understands what callers want, handles routine requests such as hours, directions and bookings, and transfers everything else to staff. You build one by connecting telephony to a real-time voice pipeline (speech-to-text, a language model, text-to-speech), giving it a few narrow tools like "check availability" and "book appointment," and designing a clean transfer to a human.
Key takeaways
- Latency and turn-taking make or break a voice agent; budget engineering time for them.
- A pipeline (STT → LLM → TTS) gives control and lower cost; realtime speech models sound more natural but cost more.
- Keep the receptionist's job narrow: answer FAQs, book, take messages, transfer.
- Always provide a path to a human, and disclose that the caller is speaking with an automated assistant.
- Per-minute costs are measurable up front; model them before you choose voices and models.
What a voice receptionist should and should not do
Good first jobs:
- Answer opening hours, location, parking, pricing ranges and services offered.
- Book, reschedule or cancel appointments.
- Take a message with name, number and reason, then create a task for staff.
- Route calls ("billing," "new patient," "existing order") to the right person or line.
Jobs to keep with humans:
- Complaints, refunds and disputes.
- Anything requiring clinical, legal or financial judgment.
- Emergencies. The agent should recognize emergency phrases and immediately give the correct emergency number or transfer.
The architecture
A voice receptionist has five layers:
- Telephony: a phone number and SIP trunk (or WhatsApp calling, or a web widget) that brings audio in.
- Real-time transport: streams audio between the caller and your agent with low delay.
- Voice pipeline: speech-to-text (STT), the LLM, and text-to-speech (TTS), plus voice activity detection and turn detection so the agent knows when the caller has finished speaking.
- Tools: functions that read and write your calendar, CRM or ticketing system.
- Observability: transcripts, recordings (where lawful and disclosed), latency metrics and error logs.
Open-source frameworks handle much of layers 2 and 3. The LiveKit Agents framework, for example, is open source under Apache 2.0, runs in Python or Node.js, streams audio through an STT-LLM-TTS pipeline, includes a turn-detection model and interruption handling, and supports SIP telephony so agents can place and receive phone calls. LiveKit's docs even include a multi-agent medical office triage example.
Pipeline vs realtime models
| Approach | How it works | Strengths | Trade-offs |
|---|---|---|---|
| Pipeline (STT → LLM → TTS) | Three separate models in sequence | Mix best-of-breed vendors, cheaper LLMs, full transcript control, easier debugging | More moving parts; latency adds up across stages |
| Realtime speech-to-speech | One model takes audio in and returns audio | More natural prosody and faster turn-taking | Usually higher cost per minute; less control over each stage |
OpenAI, for instance, lists its GPT-Live 1 voice sessions at $0.05 per minute (backend model and tool usage billed separately) on its API pricing page, as of October 2026.
What it costs per minute
LiveKit's pricing page includes a calculator. With its default selections for a phone call on the Build/Ship plans in October 2026, it estimated:
| Component | Estimated cost per minute |
|---|---|
| Agent session | $0.0100 |
| Telephony | $0.0100 |
| LLM | $0.0014 |
| Speech-to-text | $0.0075 |
| Text-to-speech | $0.0090 |
| Observability | $0.0100 |
| Total | ≈ $0.0479 |
Premium voices raise the TTS line (some listed TTS models are $0.03 per minute), and realtime speech models cost more than the default LLM line. Plan allowances matter too. LiveKit's free Build plan includes 1,000 agent session minutes, five concurrent sessions and one free US local number; Ship starts at $50 per month with 5,000 minutes and 20 concurrent sessions; Scale starts at $500 per month with 50,000 minutes and features such as region pinning and a signed BAA for HIPAA (all as of October 2026).
Two practical notes for budgeting:
- Concurrency is a hard ceiling. If five calls are in progress on a plan with five concurrent sessions, the sixth caller needs a fallback (voicemail, queue or transfer).
- Long calls cost more than you think. A three-minute booking call at about $0.05 per minute is roughly $0.15 before your own hosting and any carrier fees outside the platform.
Step-by-step build
1. Script the top call types
Listen to or log 50 to 100 real calls. Write down the five most common reasons people call and what a perfect answer sounds like. These become your test scenarios.
2. Pick models for your callers
- STT: test with real recordings of your callers' accents and any language mixing. Multilingual and medical-vocabulary STT models exist; LiveKit lists options such as Deepgram Nova-3 Medical.
- LLM: a fast, mid-sized model is usually enough. Latency matters more than raw intelligence on the phone.
- TTS: choose a clear, warm voice. Test it on numbers, dates, addresses and names, which are where voices most often stumble.
3. Write a voice-specific prompt
Voice prompts differ from chat prompts:
- Keep replies to one or two short sentences.
- Never use lists, markdown or URLs in speech.
- Read numbers in a natural way ("ten thirty in the morning," not "10:30").
- Confirm key details back ("That's Tuesday the fourth at three p.m. for Dr. Lim, correct?").
- State the disclosure in the greeting ("You've reached the clinic's automated assistant. This call may be recorded.").
4. Build narrow tools
get_available_slots(service, date_range): read-only.book_slot(slot_id, name, phone): validates the slot server-side and prevents double booking.take_message(name, phone, reason): creates a staff task.transfer_call(department): hands the caller to a person or line.
Follow least privilege. OWASP's guidance on excessive agency applies to voice agents just as much as to chat: limit tools, permissions and autonomy, and require human approval for high-impact actions.
5. Tune turn-taking and interruptions
Callers pause mid-sentence, say "um," and interrupt. Configure end-of-turn detection so the agent does not cut people off, and let callers barge in while the agent is speaking. Test on noisy phone lines; background noise suppression helps STT accuracy.
6. Design fallbacks
- If the LLM or STT provider fails, fall back to a backup provider or a simple "please hold while I transfer you."
- If the caller asks for a person, transfer immediately.
- After hours, take a message and promise a callback window you can keep.
7. Test, then soft-launch
Run your scripted scenarios plus adversarial ones (callers trying to get discounts, confused callers, silence). LiveKit and other platforms offer simulated-caller testing; whatever you use, have a person listen to recordings. Start with overflow calls or after-hours calls before you let the agent answer every call.
From experience: As CTO of Femma AI, I work on voice AI built on LiveKit. My rule of thumb from that work: callers judge a voice agent almost immediately, so a fast, natural greeting and a clear way to reach a person matter more than clever answers later in the call.
Measuring performance after launch
A voice agent that "works" in testing can still frustrate real callers. Track these numbers weekly for the first two months:
- Containment rate: calls fully handled without a transfer. Higher is not always better; a high rate with low satisfaction means the agent is trapping people.
- Transfer rate and reasons: why calls went to staff. Each common reason is either a new intent to support or a sign the agent should transfer sooner.
- Booking completion rate: of callers who wanted an appointment, how many ended the call with a confirmed slot.
- Average handle time: shorter is good for routine calls, but watch for callers hanging up early.
- Latency: time from the caller finishing a sentence to the agent starting to reply. Long pauses feel broken on a phone line.
- Misrecognition rate: names, numbers and dates captured wrongly. Review a sample of transcripts against recordings.
- Cost per call compared with your estimate.
Pair the numbers with listening. Have a staff member review ten random calls each week and tag what went well and what did not. Most improvements come from small changes: a shorter greeting, confirming phone numbers digit by digit, adding a missing FAQ, or transferring after one failed attempt instead of two.
Finally, ask staff what changed for them. If the front desk now spends its time fixing bookings the agent got wrong, the system is not saving time yet.
Privacy and compliance basics
Voice calls carry personal data, and in healthcare, sensitive health data.
- Tell callers they are speaking with an automated assistant and whether the call is recorded.
- Store only what you need; set retention periods for recordings and transcripts.
- Use vendors with appropriate data processing terms, region controls and, for US healthcare, a signed BAA where required.
- In Malaysia, the Personal Data Protection Act 2010 sets notice, security and retention principles; the Personal Data Protection Department publishes them.
This is general information, not legal advice. Check the call-recording and AI disclosure rules that apply to you.
Launch checklist
- [ ] Top five call types scripted with ideal answers.
- [ ] STT tested on real caller audio, accents and languages.
- [ ] Voice prompt keeps replies short and confirms key details.
- [ ] Greeting discloses automation and recording.
- [ ] Booking tools validate slots server-side and send written confirmation.
- [ ] Emergency phrases trigger the correct emergency guidance or transfer.
- [ ] "Talk to a person" works at any point in the call.
- [ ] Provider failover and concurrency overflow handled.
- [ ] Recordings and transcripts have a retention policy.
- [ ] Per-minute cost modeled and spend alerts set.
Next steps
- Log a week of inbound calls and tag the reasons people call.
- Price your expected minutes using a platform calculator such as LiveKit's.
- Prototype one call type (bookings) end to end before adding others.
- Reuse your chat knowledge base; see how to build an AI customer support agent and LLM API costs explained.
- Running a clinic? Pair voice with WhatsApp appointment reminders. For architecture help, a fractional CTO can scope the build.
Sources
- LiveKit pricing and cost calculator: https://livekit.io/pricing
- LiveKit Agents documentation: https://docs.livekit.io/agents/
- OpenAI API pricing (GPT-Live, realtime models): https://developers.openai.com/api/docs/pricing
- OWASP LLM06:2025 Excessive Agency: https://genai.owasp.org/llmrisk/llm062025-excessive-agency/
- Malaysia PDP principles: https://www.pdp.gov.my/en/principles-of-personal-data-protection/
Frequently asked questions
How much does a voice AI receptionist cost per minute?
It depends on the models and telephony you choose. As an example, LiveKit's pricing calculator showed about $0.048 per minute for a phone-call agent using its default model selections in October 2026, covering agent session, telephony, LLM, speech-to-text, text-to-speech and observability. Premium voices and realtime speech models cost more.
What is the difference between a pipeline and a realtime voice model?
A pipeline chains three models: speech-to-text, a text LLM and text-to-speech. It is flexible and lets you mix providers. A realtime or speech-to-speech model handles audio in and out in one model, which can sound more natural and respond faster, but usually costs more per minute and gives you less control over each stage.
Can a voice AI receptionist book appointments?
Yes, if you give it a tool that checks availability and creates bookings in your calendar or scheduling system. The tool should confirm details back to the caller, validate the slot server-side, and send a written confirmation by SMS, email or WhatsApp. Keep cancellations and reschedules behind the same validation rules.
Do I need to tell callers they are speaking to AI?
Disclosure is the safer and more trustworthy choice, and some jurisdictions require it or require consent before recording calls. Tell callers early that they are speaking with an automated assistant and whether the call is recorded, and always offer a way to reach a person. Check the rules where your business and callers are located.
Which languages can a voice receptionist handle?
That depends on the speech-to-text and text-to-speech models you pick. Several providers offer multilingual models, and LiveKit lists multilingual options from vendors such as Deepgram and AssemblyAI. Test with real callers' accents and code-switching, such as English mixed with Malay or Urdu, before you rely on it.