Real-Time Voice AI on CPU

Executive summary

A voice assistant has one constraint a chat window does not: the conversation has a clock. A pause that would be unremarkable in text is a dropped call in speech, and an assistant that talks over someone, or hears itself and answers its own voice, is not a slower chat window but a broken one. For a small business, all of this also has to run on modest hardware, with no one watching it.

This paper presents the design behind Xorventa's voice assistants, on websites with Xorventa Assist and on phone lines with Xorventa Phone Assist: how a turn is ended by silence and by whether the thought is complete, how an interruption stops the assistant without its own voice being mistaken for the caller, and how capacity is measured and enforced so that one caller too many is refused rather than degrading everyone. In our measurements a live session cost about 0.17 of a CPU core with no audible degradation from one session to six, start-up fell from over 10 seconds to 2.8 to 4.4 seconds, and sessions on our site saw a median round trip of 43 ms. We set out the problem, the design and the reasoning, along with what these measurements do and do not show.

1. The problem

Text retrieval tolerates delay and forgives interruption, because the reader is in control of both. Speech gives neither back. Three properties make voice a different engineering problem.

Nothing in the audio says a person has finished. People pause to think, and they pause when they are done. End a turn too early and the assistant cuts them off, then answers half a question. End it too late and every reply feels slow. A single silence threshold can only trade one failure for the other.

On a laptop or phone speaker, the assistant's own voice reaches the visitor's microphone. If that audio is transcribed as the visitor, the assistant starts answering itself. The obvious defence is to mute the microphone while it speaks, but then no one can interrupt it, and a person who cannot interrupt has lost the ordinary control every conversation has.

A real-time media connection keeps accepting callers past what the machine can serve. There is no queue to grow and no request to retry. When the CPU runs out, every live call breaks up together, at exactly the moment interest peaks. For a small business's assistant on shared infrastructure, the operating constraints are also tight: CPU rather than GPU, and real-time media that may be tied to a single machine, which limits how far horizontal scaling can help.

2. Why the conventional approaches fall short

3. The solution

One assistant serves text, browser voice and phone calls, answering from the same knowledge. Voice adds a real-time path around it, where each stage has a specific job and a specific way to fail safely.

The microphone is requested with echo cancellation, noise suppression and automatic gain control all on, because the browser is the only place that knows exactly what is coming out of the speaker. Recognition then uses a non-Whisper streaming engine with biasing toward the business's own terms. The engine is selected by configuration, since it may need to change. Both engines we ran emit the same interim and final transcripts, so nothing downstream depends on which is active.

The recognizer's silence window is deliberately short, at 800 ms, so that a finished thought commits quickly. When it fires, a deterministic check reads the accumulated words. If they trail off on a conjunction, preposition, article, hesitation or dangling auxiliary, with no closing question mark or exclamation, the assistant holds the floor for a further 1.5 s to let the person finish.

This resolves the conflict in Section 1.1 instead of tuning around it: the silence threshold is free to be short because a separate signal protects the person who is still thinking.

Interruption depends on four independent gates: the pipeline must allow it, the speaking signals must reach it, a transcript arriving during playback must not be dropped, and the browser must be told to clear audio it has already buffered. Without that last gate the server stops generating and the visitor still hears the rest of the sentence. Any one gate left closed silently disables the others, so one setting drives all four.

Two checks keep the assistant's own voice from starting a turn. They run before the interruption gate, so opening it does not open a path for echo: a transcript that appears verbatim in the last reply, or that shares at least 70% of its words with it, is treated as echo, within a recency window and with a guard for real questions. An interruption must also reach two words, so a cough or a stray "uh" does not stop the sentence.

A WebRTC jitter buffer is empty at the start of each reply, and if the first audio outruns it, the listener hears a hitch in the first word. A 200 ms leading silence fills it first. The injection is anchored to the single start-of-speech event for the turn and never inferred from a gap in the audio, because a late chunk mid-sentence would then be mistaken for a new reply, and silence spliced into a word is an audible click. Spoken replies are limited to three sentences, because a voice reply that runs long cannot be skimmed.

Much of the wait from pressing the button to hearing a voice is not in the server. A browser's first peer connection in a tab took 3.5 to 4.3 s, and every later one took 0.003 s. So the widget pays those once-per-tab costs when the visitor opens the window, not after the press. It also opens the microphone once, holding the permission check's stream until the call connects, and waits at most 0.6 s for connection candidates, letting late ones arrive afterwards.

Voice retrieves from the same knowledge as chat, using the same fact records and retrieval described in our paper on grounding AI chat, at the same cost of milliseconds per turn. Retrieved text reaches the model between delimiters with a per-turn random nonce, as data. Spoken answers sound more certain than written ones, which is a reason to keep this path the same as chat rather than build a second one. A prompt-cache warm-up during the greeting keeps the first real turn from paying the full prompt cost.

A call should be able to end because it is over, not only because a timer ran out. The two call-ending actions are deliberately opposite:

Hard limits still exist for abandoned sessions: 15 minutes in total, and two minutes without audio in either direction, or one minute on a phone line.

A platform-wide limit applies across all customers, because the exhausted resource is the machine. Browser and phone sessions draw from the same count. Past the limit, a new caller is told at once and offered text chat, and calls already in progress are not disturbed. The limit is set from the measured cost per session and kept below saturation, allowing for the fact that one interpreter process may not be able to use every core. The operator is alerted before the limit is reached, ahead of anyone being refused.

4. The evidence

Frames arrive every 20 ms, and a gap above 60 ms, three frame periods, is counted as a glitch because it is audible as break-up. The media and pipeline runs were August 2026.

Dividing a core by the cost of one session understates capacity, because a single session carries all the fixed overhead. Dividing it by the media-only cost overstates capacity. Only the full pipeline at several levels gives the number that matters.

A single earlier run had given a multiple of 1.69, which would have made a planning assumption of 2 look cautious. It did not reproduce: repeated runs came out 25 to 40% above that assumption. The two engines overlap, so the engine is not what decides capacity.

From pressing voice to the first word, on a microphone already permitted, the design in Section 3.5 reduced the time from 10.3–11.0 s to 2.8–4.4 s, measured with a real microphone against the live service on one laptop.

5. How this behaves in a call

The result is a clear division of labour. Decisions in the call's path are deterministic and take no time: completeness, echo and admission. The time that remains is spent where it improves the answer, in the model and the voice.

6. Guidance for teams building similar systems

7. Conclusion

Most of what makes a voice assistant feel right is not the model. It is a set of small, deterministic decisions about whose turn it is, whose voice that was and whether another caller can be served well, with each one designed to fail in the direction that costs the person least. These can be built, tested and reasoned about without a GPU.

These results come from our own engineering tests and from sessions on our own service, within the scope in Section 4.6. The methods transfer directly, and we encourage teams building real-time voice to run them on their own stack.