How we decide a voice agent sounds more natural.

By October our agents replied fast. Then Sneh listened to two calls from 12 October on a Gujarati appointment line and said they still didn’t sound like a conversation.

In the first, the caller answered a question with “ના, નથી.” (“no, I don’t”). Then nothing happened for 9.7 seconds. The caller said “હેલો.” and the agent read out its opening menu, as if the call had just started.

The second problem was repetition. Across both calls, two words, સમજ્યું (“understood”) and સરસ (“great”), opened 20 of the agent’s 31 turns. In a study of human task dialogues in English, 25.9% of turns start with a word like that.

His bar was rhythm, listening while the caller speaks, acknowledgments that fit, and no repetition. Building the fixes was the easy part. Most of the work went into deciding what would count as proof.

We made six changes, each behind its own switch.

Each one went in behind its own switch in the worker.

1. A dead-air watchdog. The 9.7-second hole had a dull cause. 52 ms after the caller’s last word was transcribed, half a second of sound with no words in it opened the voice detector and cancelled the reply already on its way. Nothing ran it again. Now the worker re-runs a reply that a wordless sound cancelled. It also resumes the last question when a caller checks in with “hello?”, varies its opening acknowledgment, and knows Gujarati backchannels, so a caller’s “હા હા” mid-sentence no longer stops the agent.
2. An adaptive turn floor. It waits longer while someone reads out a number or an address, and replies sooner after a plain yes or no.
3. A listener that can say “હં” or “हम्म” while the caller talks. It ran in shadow mode, only logging where it would have spoken.
4. A short note sent to the model before every reply: the facts the caller has already given, the openers used recently, one question at a time, and a word-for-word repeat when the caller says “huh?”. This one comes from the literature. In one study, telling a model to “be diverse” moved its reuse of the same move from 0.57 to 0.52. Showing it the moves it had already used took it to 0.40.
5. A prompt cleaner. People paste in prompts written for a chat window, with headings, tables, emoji and lists of phrases to say. The worker rewrites the copy the model sees into spoken instructions, and marks phrase lists as examples of tone only. The stored prompt never changes.
6. Better goodbyes. One goodbye, and no “shall I hang up?” after the caller has already said bye.

Each round dials two real workers at once.

A harness dials two real workers at the same moment over the browser call path: main, and main plus the changes. Both use the production engines and hear the same synthetic caller.

The scenarios are the moments we kept hearing go wrong: the “no”, silence, “hello?” sequence; an address read in two parts with a pause after “और”; a pincode in groups of digits; a Gujarati “હા હા” one second into the agent’s reply; a complaint; ten short answers in a row; an 8.4-second monologue with two pauses; a polite close. Each runs three times per arm. Round four was 87 calls.

We score in three layers.

First, plain counts, with no model involved: silences over 1.5 seconds after the caller finishes; restarts, where the agent repeats its greeting or menu after the first turn; repeated sentences; the share of replies that open with an acknowledgment, or reuse one from the last three replies; and turn splits, where the system took a caller’s sentence before they’d finished it.

Second, blind judges. Each pair of calls, same scenario and same repetition, goes to an LLM as two transcripts labelled X and Y at random. From round three on, the transcripts show response gaps, silences, overlaps and cut-offs. The judge scores rhythm, listening, affirmation, continuity, repetition and closing from 1 to 5. By round three there were three judges from three model families: Gemma, which is also the model the agent runs on, gpt-oss-120b, and Claude Opus.

Third, a person with headphones. Each pair plays as stereo audio, caller on the left and agent on the right, in random order. The key opens only after scoring.

The first three rounds all said no.

Round one lost. The judge preferred main in 51 of 81 judgments and the branch in 26, and every loss traced back to a log line. The check-in fix played “હા, હું અહીં જ છું.” (“yes, I’m here”). The model then re-asked its question word for word, an older repeat guard dropped that, and the regenerated reply echoed “I’m here” again. The number rule held 0.8 seconds on any answer to a question that mentioned a number, a bare “હા.” included. On the 26 turns where it fired, the branch was 493 ms slower than main on average, and only 6 of those turns needed the wait. The listener’s sounds landed just after the caller had finished.

We fixed those. Round two came out level, 36 judgments to 35.

Round three went backwards, because two of our own changes disagreed. The turn floor read “ऑर्डर नंबर चार पाँच दो एक” (order number 4521) as a complete number. The per-turn note had its own rule, that under ten digits after “नंबर” meant unfinished, so it told the model to answer with a short “जी..” and wait. The voice engine drops inputs that short. The watchdog saw a reply on its way and stayed quiet, and the caller sat through 4 to 6 seconds of silence. Now any note that asks the model to wait counts as a dead-air risk unless the turn floor agrees.

In round four, the branch led on every judge.

For the first time, the branch was ahead on every judge. Claude Opus gave it 24 of the 33 scripted pairs, gave main 3 and called 6 even (sign test p < 0.001). Gemma split 16, 9 and 8 (p = 0.23), and gpt-oss-120b 15, 7 and 11 (p = 0.13). Only the Opus result is decisive on its own.

The counts moved more clearly than the judges. Per call:

measure	main	branch
restarts	0.64	0.24
agent cut off by the caller	0.18	0
repeated sentences	0.36	0.03
turn splits	1.61	1.06
opener reused within three replies	10.7%	0%
replies opening with an acknowledgment	50%	40%
bigram self-repetition	0.130	0.085
silences over 1.5 s	0.03	0.12

The last row went the wrong way, and so did latency. Replies were 62 ms slower on average, and the 95th percentile moved from 1,124 to 1,410 ms. Most of that tail was one scenario. A wordless hum right after the caller’s “no” restarted the turn floor, and the reply landed 1.8 to 2.1 seconds later: the original bug again, smaller. Sneh had said a little latency is fine if it buys naturalness, so we took the trade, and fixed the hum before merging. A sound with no words no longer moves the floor clock. We haven’t re-run the full A/B since, so the numbers above are round four’s.

One scenario used a prompt with two lines of the kind people paste in: “acknowledge briefly” with two example words, and a scripted menu line for when the caller says hello. Main said the menu line after “હેલો.” in 3 of 3 runs and opened every reply with one of the two example words. The branch never said the menu line and never reused an opener within three replies. But it still opened 78% of its replies with some acknowledgment, so how often it acknowledges is only half fixed.

Then Sneh listened to 15 blind pairs from round four: 7 for the branch, 5 for main, 3 the same. His word was “slightly better”. The pairs he gave main came down to the wait on digits and a double goodbye after the caller said thanks. Both were fixed after the round.

These are the results we refused to count.

A single judge. In round three the judges barely agreed, with Cohen’s κ between 0.07 and 0.45. On the scripted-menu scenario, Gemma scored main’s restart 4.83 out of 5 and Opus scored it 2.67. Gemma also runs the agent, so it was judging its own calls. Agreement rose in round four, but no single judge decided anything.

Small differences, and scores across rounds. With three runs per scenario, one event in three calls is noise, so a count had to move by more than 0.34 per call before we called it a change. The stack, the transcript format and the scenarios changed between rounds, so we compared directions across rounds, never absolute scores.

Replays of real calls, taken at face value. We streamed the caller’s side of the 12 October recordings into both arms. But a recorded caller is answering the original agent, so once ours is a step out of order, every later answer is too. Opus gave all three pairs of one replay to main. Part of that was the replay. Part was a real bug: a continuer too short for the voice engine to speak, which left 6.6 seconds of silence.

Our own counters. The watchdog counted that dropped continuer as said. We caught it because the harness scores what the caller would have heard, rebuilt from a model of the browser’s audio player, rather than what the worker sent.

All six changes are now merged.

All six changes are merged. The listener stays in shadow mode by default, because its one remaining sound in round four still landed just after a long monologue ended. Next we read the same counters on real production calls before turning the listener on for one agent. And we’re training an end-of-turn model for Gujarati, which the literature puts ahead of any silence rule we could write.