One prompt change broke 19% of our test calls

On June 11 a customer of ours, a scheduling platform that runs voice agents for about 140 dental clinics, pushed a prompt edit to staging. The diff removed 14 words and added 9. Forty minutes later the Sonora deploy gate blocked the release: task success across their 620-call scenario suite had fallen from 94.2 percent to 75.1 percent, and 118 calls that passed the day before now failed. That is 19 percent of the suite, broken by a change that read like a copy tweak.

This is the post-mortem. We are publishing it because the failure pattern is common, quiet, and nearly impossible to catch by listening to a handful of calls.

The change looked harmless

The clinics had one consistent complaint: the agent talked too much at the top of a call. Greetings averaged 11.4 seconds, callers talked over them, and front-desk staff wanted the agent to get to the point. An engineer tightened the system prompt:

“Be brief. Do not repeat information the caller already has.”

He listened to five staging calls afterward. Greetings dropped to 4.2 seconds and the agent sounded sharper. Review approved the change in under ten minutes. Two years ago, this ships to production the same afternoon.

What actually broke

The agent stopped reading confirmation numbers back to callers.

The mechanism is worth understanding. When the booking tool returns, the confirmation number lands in the agent’s context. From the model’s side of the conversation, that number is now old information. Under the new instruction it classified the readback as repetition and skipped it. The caller, of course, had never heard the number at all.

Nothing crashed. Latency was fine. Accuracy was fine, since everything the agent said was true. The failure was a sentence that no longer existed, and you only find a missing sentence by checking the transcript against a rubric that expects it.

The failing turn

Scenario 217 in the customer’s suite is a reschedule flow replayed from a real call. Here is turn 6 on the baseline build and on the candidate:

In production that missing sentence becomes a callback. Clinics ask for the reference at check-in, and this customer had measured the cost before: a readback outage the previous quarter produced 63 extra front-desk calls in two days.

What the regression alert showed

The customer’s gate blocks any candidate that drops suite task success by more than 2 points against the running baseline. Here is the report it posted to their Slack channel:

Two details in that report mattered more than the headline number. First, tone went up. The agent sounded better while doing less of its job, and the five calls a human reviewed were sampling exactly that improvement. Second, accuracy barely moved, because accuracy checks what the agent said and this failure lived in what it did not say. Only the task-success rubric, which requires an explicit readback on booking turns, registered the break.

Turn-level scoring did the diagnostic work. The failing calls still passed 8 of their 9 turns, so a call-level average would have read about 89 percent and looked like noise. Scored turn by turn, 118 failures collapsed onto a single rubric item, confirm_reference, and the alert could name the exact behavior that disappeared.

The fix took one line

The team kept the brevity instruction and added one sentence: “Always read the confirmation number back, digit by digit, even when being brief.” The rerun scored 94.6 percent on task success with greetings at 4.6 seconds, better than baseline on both counts. Elapsed time from blocked gate to green rerun: 41 minutes. The regression never left staging.

What we tell every team now

• Test on real transcripts. Scenario 217 exists because a real caller once asked to move a cleaning and then waited for the number. Synthetic happy-path tests rarely encode that expectation, and this bug passes every one of them.
• Gate deploys on scores, not listening. A human heard 5 calls and approved. The gate ran 620 and blocked. Prompt edits are code changes and deserve the same CI treatment.
• Score turns, not calls. The call-level view of this incident was an 89 that looked like variance. The turn-level view was 118 failures on one rubric item, which is a diagnosis.

A diff that touched 23 words cut task success by 19.1 points. If your agents talk to customers, put a suite between the prompt and production, and make sure that suite has heard your real calls.

Incident details are an anonymized composite of regressions Sonora has caught for customers, shared with permission.