Same accuracy on automated decisions. One‑fiftieth of the cost.
DW Daniel WipertWe put a small, specialized decision model, Jev, and two general-purpose models, Claude Sonnet 5 and Claude Haiku 4.5, into the same automated triage pipeline. On the customer messages each model was confident enough to handle on its own, Jev was as accurate as Sonnet 5, at about 2% of the cost.
Most business AI isn't writing essays. It's sorting: which team should get this message, is this refund request valid, what kind of ticket is this. We tested that kind of work.
Each model read a real bank customer message and picked one of 77 categories, such as "card hasn't arrived" or "lost or stolen card", along with how confident it was.
Simple checks ran first. Then only confident answers were accepted automatically, and the rest went to a person. We measured how often each model was right on what it accepted.
The 1,000 test messages were locked away until the end, and the rule for "who wins" was written down before the first run.
Each finding has the short version first. Open "Show the numbers" for the evidence, or read the technical report for the full method.
Set to auto-accept about 70% of messages, Jev was right 91.4% of the time and Sonnet 5 91.0%. That difference is within noise: call it a tie. Haiku 4.5 trailed at 84.6%.
The chart shows every setting a team could choose, from "automate a little, very safely" on the left to "automate almost everything" on the right.
Accuracy on auto-accepted messages at each pre-registered setting. Each model's cutoff was chosen on a separate 500-message tuning set, then applied unchanged to the test set.
Why "tie" and not "win": with about 730 accepted answers per model, each accuracy figure is uncertain by roughly ±1 point. We also compared each model's 730 most confident answers directly: Jev %, Sonnet %. Still a tie.
Jev cost $0.10 per 1,000 decisions. Sonnet 5 cost $5.25, 52 times more. Even the smaller Claude model, Haiku, cost 19 times more than Jev, while being less accurate.
Costs are what OpenRouter billed for each call, logged per message. Jev charges only for input. Claude calls were one request per message with extended thinking off.
Let Jev handle the messages it's sure about and pass the rest to Sonnet 5. That cascade answered every message with 82.8% accuracy for $1.51 per 1,000. Sonnet on its own: 82.3% for $5.25.
The best system wasn't a single model. It was a routing design.
The cascade at each of Jev's settings. It needed no extra calls: it was computed from the Jev and Sonnet logs.
Sonnet's share looks weak (about 60% right) because it gets only the hardest messages, the ones Jev wasn't sure about.
Asked how sure they are, the Claude models mostly answer 85, 90 or 95. So many answers share the same score that a team can't set the automation level precisely. Jev's native scores are finer-grained.
At the "70%" setting, Jev actually auto-accepted 73% of messages, Sonnet 75%, and Haiku 81%.
Calibration asks: when a model says "90% sure", is it right 90% of the time? The gray line is perfect calibration. Sonnet 5 is the most trustworthy when it's confident. It just gives fewer, coarser scores.
Before trusting any confidence score, the pipeline runs simple Python checks. A keyword check ("does the message mention anything related to the chosen category?") stopped about 1 in 7 wrong answers while blocking under 2% of right ones.
And no invented category was ever auto-accepted. Jev can't produce one by design. Haiku produced two malformed replies, and the gate stopped both.
The keyword lists were generated automatically from training messages (not the tuning or test messages). The check started as a logged flag only and was switched to blocking after the first tuning run showed a 1.5% false-alarm rate. That decision is logged in the changelog.
We rewrote the descriptions of the most-confused categories. That added about 1 point for Jev, and the same for Sonnet, so the rewrites didn't favor Jev.
A second round of rewrites gained nothing. And a 3.6-point lead Jev showed during tuning vanished on the locked test set, which is exactly why we held it back.
"Before" uses descriptions that are just the category names. "After" uses descriptions rewritten to fix Jev's most common confusions: drafted by Claude Sonnet 5, approved or edited by a person. The Sonnet "before" run was added specifically to test whether this favored Jev. How it would be reported was committed in advance.
We then tried DeepSeek V4 Flash, one of the cheapest capable models on OpenRouter, under the same pre-registered rules. It cost 8× less than Jev, but at the 70% setting it was right 88.0% of the time, against Jev's 91.4%.
Compared at the same volume, that's about more wrong answers per 1,000 going out with nobody checking them, to save 9 cents.
DeepSeek's confidence is the probability it gave the category it wrote (from log-probabilities), the same kind of score as Jev's. Its cutoffs came from its own tuning run, as for every model. The rules were written down before the full run.
Same volume check: each model's most confident answers. Jev %, DeepSeek %.
Overconfident scores: DeepSeek put nearly half of all messages at 99.9% sure or higher, and only about 94% of those were right. Its high scores are less trustworthy than Jev's, which is what matters when the score decides what gets automated.
When Jev was confidently wrong, DeepSeek usually picked the exact same wrong category (). That points at the answer key, so we re-checked 206 doubtful labels, blind to which model said what, and published every verdict.
On a corrected key every model scores higher, but Jev stays ahead of DeepSeek at every level of automation. Fixing clearly wrong labels barely changes the gap. Accepting either label where both fit shrinks it, because DeepSeek is very confident on ambiguous messages and Jev isn't. Of DeepSeek's 45 extra mistakes, 28 are on messages nobody disputed.
How much of the gap is the labels? Jev minus DeepSeek, comparing each model's n most confident answers, on each key. The range is a 95% bootstrap interval over the test messages; with one run per model, it doesn't cover run-to-run variation.
Which labels were checked: the 206 test messages where at least 2 of the 4 models (Jev, Sonnet 5, Haiku 4.5, DeepSeek) agreed on a different category than the answer key. Each was shown with the two labels in random order, without saying which was official. Verdicts: original kept , changed , both fit , unclear (original kept). See every label →
All four models at the 70% setting:
Read with care: this is a secondary result; the official numbers above don't change. The verdicts were drafted by Claude and approved by the author in bulk, not row by row. Only messages where models agreed against the key were reviewed, which favors "popular" answers. The re-scoring rule was written down before any corrected score was computed; the clear-errors-only key, the full curves and the gap intervals were added afterwards.
Leaderboards rank overall accuracy. In production you only automate the confident part, and there a small specialized model can match a frontier one. Sonnet was slightly better on all answers (82.3% vs 81.6%), which would have picked the wrong model.
Your automation rate and your error rate both come from the confidence score. A model whose confidence can't be tuned finely is a model whose error rate you can't control precisely.
Rule-based checks, a confidence bar, and a cascade to a stronger model did as much as the model choice. Because the pipeline stayed fixed, swapping models was a one-line change.
Locking the test set, writing the rule before the data, testing our own weak spot, and publishing every log. The tuning-set lead that disappeared shows why: the undisciplined version would have been wrong.
Every number on this page is computed from the public decision logs. You can re-run the analysis without making a single API call.