How Sonora scores a call

Every number Sonora produces can be traced to a turn, a rubric item, and a span of transcript. That is the design constraint the whole scoring pipeline hangs off, because a score you cannot explain is a score you cannot act on. This note walks through how a call becomes a grade, using the same pipeline that grades 500,000+ calls a month for 120 voice teams.

Three dimensions, scored separately

A call gets three scores, not one, because the three ways an agent fails are independent and demand different fixes.

Accuracy

Every factual claim in a turn is checked against a source of truth: the tool outputs in the trace, the knowledge base the agent was given, and what the caller actually said. An agent that quotes a $180 refund when the tool returned $130 fails accuracy, however warmly it says it.

Tone

Register, pacing, and behavior under pressure. Does the agent stay level when the caller is angry, yield when interrupted, and avoid reading like a policy document out loud. Tone problems rarely fail a task today; they raise handle time and churn next quarter.

Task success

Did the turn move the caller toward what they called for, and did the call complete the steps that matter: identity verified, the correct slot booked, the reference read back. Task success is defined by rubric items, not by whether the conversation ended politely.

Averaging these into one number destroys the diagnosis. A perfectly polite agent that books the wrong day scores high on tone and fails task success, and the fix depends entirely on which one broke.

Turns, not calls

If a score cannot point to the turn and the rubric item that produced it, it is a vibe, not a measurement.

Sonora grades every agent turn independently. The grader is a language model, but a constrained one: it never emits a freeform 1 to 10, because ungrounded scales drift between model versions and between Tuesdays. It walks a rubric of binary and weighted criteria and must cite the transcript span that justifies each verdict. A verdict without a citation is discarded and regraded.

Here is why the turn matters more than the call. Consider a 22-turn refund call where exactly one turn goes wrong:

Averaged, this call scores 93 and looks healthy. Ranked by call-level score, it never surfaces. The caller still had a wrong number quoted at them and still called back the next day. Turn-level grading puts turn 14 in a queue with every accuracy failure like it, so the pattern is visible after 5 calls instead of after 5,000.

The rubric is the spec

A rubric is a short, versioned list of weighted criteria for a given moment in a call. Here is a trimmed example for a refund confirmation turn:

Two properties keep grades honest. Criteria are concrete enough that two engineers reading the same transcript would judge them the same way. And weights encode priorities: getting the amount right matters three times as much as the courtesy close, and a failed high-weight item fails the turn regardless of the arithmetic.

Calibrated against 25,000 human labels

A grader is only worth what it agrees with. We maintain a calibration set of 25,000 production turns labeled by trained human raters, with a 6,000-turn subset labeled independently by three raters each. On that set the grader matches the human majority label on 93 percent of turns; the humans match each other on 95 percent. The gap between the grader and the human ceiling is about two points, and we publish it rather than hide it.

Calibration is not a one-time event. Every grader model change reruns the full set, and a rotating audit slice runs weekly. If agreement on a rubric drops below 90 percent, grading for that rubric pauses and pages our eval team before it pages you.

Latency and interruption signals

Transcripts miss half of what a caller experiences, so the audio is measured too. Per turn, Sonora records response gap at p50 and p95, dead air longer than 2.5 seconds, talk-over ratio, and barge-in recovery: when the caller interrupts, does the agent stop, and does it resume without repeating itself. These land as signals attached to the turn. Teams often promote them into rubric items, for example failing any turn where the caller waits more than 4 seconds for an answer to a yes-or-no question.

Custom rubrics

The built-in rubrics cover scheduling, support, and sales flows, but the expensive failures are domain-specific. Teams write their own items in plain language, assign weights, and version them like code. When a rubric changes, Sonora replays your call history against the new version, so you can compare v4 to v5 on the same 10,000 calls before either one gates a deploy. On the Scale plan, a dedicated eval engineer helps translate policy documents into items that grade cleanly.

The output of all of this is boring by design: when a score moves, you get the turn, the item, and the diff that moved it. For what that looks like on a bad day, read the post-mortem: one prompt change broke 19% of our test calls.

Volume and calibration figures in this note are rounded for clarity.