Voice agent

What is a voice agent?

A voice agent is software that holds a spoken conversation. It listens while you talk, works out what you want, looks something up or writes something down, and answers out loud, over the phone or through a microphone in an app.

That is the difference with the phone menu everybody already knows. An IVR menu makes you choose from a list and press a digit. A voice agent lets you say "I called yesterday about order 40218, has it shipped yet" and gets on with it. OpenAI sums up what its Realtime API is for in one line: speech-to-speech agents that listen, reason, speak, and call tools.

Technically it is a chatbot with a microphone attached, but a caller will not wait for a spinner, cannot reread an answer, and will talk over the agent mid-sentence.

A pipeline or a speech-to-speech model?

There are two ways to build one, and the dimension that separates them is latency against control.

The pipeline chains three components you already know. Speech-to-text writes down what the caller said, a language model decides what to answer and which system to query, text-to-speech reads the answer out. Microsoft's Copilot Studio calls its version a text LLM voice model and describes exactly that sequence. A speech-to-speech model replaces all three with one model that takes audio in and gives audio out, which is what Microsoft's real-time voice agents and Google's Live API run on.

Speech-to-speech wins on time, because nothing waits for a finished transcript and there is no second handover to a synthesiser. It also keeps what a transcript throws away: hesitation, emphasis, the fact that the caller is annoyed.

The pipeline wins on everything you want to inspect afterwards. You have the exact text at every step, you can check an answer before it is spoken, and you can pin each part to a region. OpenAI's documentation shows where the seam is: the realtime model does not have enough information to precisely align transcript and audio, so cutting a response short removes the transcript of the unplayed portion. What lands in your logs is not exactly what the caller heard.

Region can decide the question for you. Microsoft writes that since July 2026, EU Data Boundary customers can run its text-based voice model with in-region processing but cannot use its GPT-Realtime models, because those require cross-geo processing.

Where the time goes

Researchers at the Max Planck Institute for Psycholinguistics compared conversations in ten languages and found the same pattern everywhere: people avoid talking over each other and keep the silence between turns as short as they can, with the average gap per language staying within about 250 milliseconds of the cross-language mean. Everyone who phones you has spent a lifetime inside that rhythm, so a second of dead air after a question reads as the other party not having understood.

The time goes to more places than people expect. Deepgram's architecture guide budgets 200 to 400 milliseconds for SIP and PSTN overhead before any processing starts. Then the system has to decide the caller has finished, then the model has to begin answering, then speech has to be synthesised. ElevenLabs quotes about 75 milliseconds for its Flash models but says plainly that this is model inference time only.

So no step may wait for the previous one to finish. The model streams its answer token by token, as the token streaming entry describes, and the synthesiser speaks the first sentence while the model is still writing the second. Three finished steps add up every wait; three overlapping streams cost you only the slowest.

The hard parts nobody puts in a demo

Knowing when the caller has stopped talking. The classic approach waits for silence, which OpenAI offers as a server VAD mode with a tunable silence duration; Google recommends allowing at least 500 milliseconds so the system tolerates natural mid-sentence pauses. Too short and the agent interrupts someone thinking about their order number, too long and every turn drags. OpenAI's semantic VAD listens to the words instead of the gap and works out from what was said whether the speaker has finished. Noise pulls the same dial the other way: OpenAI notes that a higher activation threshold needs louder audio and may work better in a van or a workshop, which costs you the caller who speaks softly.

Being interrupted. A caller who hears the wrong answer starts talking straight away. Google's Live API cancels and discards the generation in progress and flags the interruption so your application can stop the audio, but the agent still has to pick up from what the caller actually heard rather than from what it had written.

Accents, and Flemish against Dutch. A model trained mostly on Netherlands Dutch handles a West Flemish caller worse, and handles your product names, street names and municipality names worse still. Do not judge that on a vendor demo: take twenty recorded calls from your own line and run them through first.

Names, addresses and numbers. This is where most pilots quietly die. Order numbers, IBANs, a house number with a letter, a surname nobody spells the obvious way, a date said as "next Tuesday". Let the caller type digits on the keypad, which is why platforms like Copilot Studio still take DTMF input, and have the agent read the value back before it acts on it.

What to give a voice agent, and what to keep away from it

The calls worth automating share a shape: the answer already sits in a system, and being wrong is embarrassing rather than expensive.

1. Opening hours, stock and order status after hours. The caller wants one fact and your ERP has it. At eight in the evening the alternative is your voicemail.

2. Qualifying and routing. Work out what the call is about and who should take it, then transfer with the context attached.

3. Taking a callback request. The agent notes a name, a number, the subject and a moment that suits, so nobody sits in a queue.

4. Confirming and moving appointments. The agent phones out, the call is short, and it saves you a no-show.

What it should not get is anything where being wrong has a price. A complaint needs someone with the authority to put it right, and a customer explaining to a machine why they are angry gets angrier. Anything contractual, a price, a delivery promise, a cancellation, becomes a commitment your company then owes. And anything where a wrong number costs money needs a person to confirm it, because the agent has no idea it misheard.

None of it works without the plumbing: a number, which in the Microsoft stack comes from Azure Communication Services and reaches the agent over PSTN or SIP, read access to the system holding the answer, a calendar it may write to, and a route back to a person that carries the transcript along. Design that handoff first rather than last, because the moment a caller asks for a human is the moment your automation either pays for itself or costs you the customer.

Telling the caller, recording the call, and keeping the file

Say that it is a machine. Article 50 of the EU AI Act has applied since 2 August 2026 and asks that people know they are dealing with an AI system unless that is obvious. Spoken, that means the agent introduces itself as an automated assistant before the substantive part of the call. A human name and a natural voice with no such line is exactly what the obvious-AI exemption does not cover. The transparency obligation entry works this out in detail.

Recording is a separate question, and Belgium is stricter than the GDPR alone. The Belgian Data Protection Authority points at Article 124 of the Act on electronic communications: nobody may deliberately take note of information transmitted electronically without the consent of all parties involved. The parties have to be told before the recording, of the recording and of its precise purpose, and the authority says explicitly that this covers customers and third parties on the line, not only your own staff.

Audio, transcript and the text you derive from it can end up in three different places. Decide which you actually need, write down how long you keep each, and check where your vendor stores them.

How to tell whether it works

Containment, or deflection. Microsoft defines it as the share of requests handled in self-service that a person would otherwise have taken. Every vendor quotes it, and you should read it last.

Resolution and escalation rate. Read them together, and mind the trap in Microsoft's own definition: a session counts as resolved when the caller is asked whether that answered their question and either says yes or does not respond at all. Someone who gives up and hangs up lands in the same bucket as someone who was helped. Keep abandoned sessions separate, and treat a rising resolution rate next to a rising abandon rate as bad news.

Average handle time and transfer rate. An agent that answers instantly but transfers seven calls in ten has added a step to your process. A transfer is fine when the agent gathered something first.

None of those numbers tells you whether the caller got what they phoned for. The only thing that does is listening. Take ten calls a week, start to finish, including the ones the dashboard scored as resolved. That is where you find the agent's weak spots and the process problems your phone line has been hiding for years.