Technical report · September 2026

Jev vs. the Frontier: method and full results

DW Daniel WipertFounder, Chorus AI Systems

The overview gives the short version. This page documents everything needed to check or reproduce each number. All tables are generated from docs/data.json, which build_site_data.py builds from the committed decision logs.

Summary

We held a Python decision harness fixed (decide → gate → check → learn) and varied only the model in the "decide" seat: TypeSafe's Jev (typesafe/jev-1.13), Claude Sonnet 5 and Claude Haiku 4.5. The task was single-label, 77-way intent classification on Banking77. A four-check gate, deterministic checks first and model confidence last, decides which answers are auto-accepted.

Under a criterion written before any runs, Jev's accuracy on auto-accepted messages at the 70% auto-accept setting was 91.4%, against Sonnet 5's 91.0%, at $0.10 vs $5.25 per 1,000 decisions. Both pre-registered conditions (within 3 points; at least 10× cheaper) were met. A cheap-first cascade (Jev, falling back to Sonnet) reached 82.8% accuracy on all messages at $1.51 per 1,000, against Sonnet alone at 82.3% and $5.25.

Task and data

Models and access

Jev question

Choice(instructions="Which category best describes this bank customer's message?",
       criteria={category_name: description, ...77 entries})

Claude prompt (identical for Sonnet and Haiku)

Classify this bank customer's message into exactly one of the categories below.

Categories (name: description):
{77 lines of "- name: description"}

Message:
{message}

Reply with JSON only, in this form: {"category": "<exact category name>", "confidence": <0-100>}
"category" must be copied exactly from the list. "confidence" is how sure you are, 0 to 100.

Markdown code fences around the JSON (which Haiku sometimes adds) are stripped before parsing. Any reply that still can't be parsed is logged verbatim as the predicted category, so gate check 1 counts it as a hard hallucination.

Harness and gate

The gate (gate.py) runs four checks in order. The first failure sends the message to review, and the reason is logged.

  1. invalid_category: the prediction must exactly match one of the 77 names. Jev can't fail this by construction.
  2. empty_input: the message must be non-empty after stripping whitespace. It never fires on Banking77; it exists so the gate can be reused unchanged on email.
  3. keyword_miss: the message must contain at least one keyword of the predicted category, matched as lowercase whole words ([a-z]+).
  4. low_confidence: confidence (0–100) must be at or above the model's threshold for the chosen setting.

Keyword lists

Generated by build_categories.py: the words in the category name (minus filler such as "or", "not", "why"), plus the 10 most distinctive words per category. "Distinctive" means the highest ratio of in-category document frequency to overall document frequency, among words appearing in at least 5% of the category's messages, with ties broken alphabetically. The lists were learned from the 9,503 train messages not in the tuning set, so check 3 isn't graded on its own source data.

Check 3 ran as a logged flag during the first tuning run. It flagged 1.5% of correct answers and 14.3% of wrong ones, below the pre-stated 10% false-alarm rule of thumb, so it was switched to blocking before any other runs. Because analyze.py re-gates every logged answer, all results reflect the blocking gate.

Threshold procedure

A "90" from Jev and a "90" from Claude don't mean the same thing, so each model gets its own thresholds:

  1. Run the model on the 500-message tuning set and apply gate checks 1–3.
  2. Sort answers that pass checks 1–3 by confidence (descending, ties by message_id).
  3. The thresholds are the confidence values at which 50%, 70% and 85% of all tuning messages would be auto-accepted.
  4. Apply those thresholds unchanged to the test set, and report the realized auto-accept rate alongside accuracy.

Because a threshold can't split tied confidence values, realized rates overshoot the targets. This matters most for the Claude models, whose self-reported confidences cluster on round values (see robustness).

Description tuning (the Learn stage)

v1 descriptions were the category names with spaces. Each tuning round: analyze.py finds the top 8 confused pairs in Jev's tuning log, draft_rewrites.py sends the pairs and 3 example errors each to Claude Sonnet 5 in one call, a human approves or edits each draft, and the result is saved as a new version with a changelog row.

All drafts, approved versions and prompts are committed under drafts/. Drafting costs are logged too, including an estimate for three early calls that failed before per-call logging existed. Disclosure: description drafts were written by Claude Sonnet 5 and approved by a human.

Pre-registered criterion

Copied verbatim from the project spec, written before the first run:

Jev wins if, at the 70% auto-accept row on the final test set: its accuracy on auto-accepted messages is within 3 points of Sonnet 5's, and its cost per 1,000 gated decisions is at least 10x lower than Sonnet 5's.

The 50% and 85% rows are supporting evidence: they show whether the gap widens or narrows as each model is pushed into its less-confident answers.

"Not yet" if Jev misses either condition. The headline then reads "not yet the right gate for this task," and the cascade (contender E) is reported alongside as the practical alternative.

Outcome: Sonnet − Jev = −0.4 points (pass); Sonnet costs 52.5× more (pass). Jev wins.

Results

All contenders, test set (1,000 messages)

A: Jev, v1 · B: Jev, v2 (final) · C: Sonnet 5, v2 · D: Haiku 4.5, v2 · F: Sonnet 5, v1 (tilt check) · J: DeepSeek V4 Flash, v2 (round 2) · K: DeepSeek V4 Flash, v1. Soft hallucination = 1 − accuracy on auto-accepted. Hard hallucination = share of all answers failing check 1. The highlighted rows decide the verdict.

Cascade (E): Jev if auto-accepted, otherwise Sonnet 5

Computed from the B and C logs, with no additional calls. Cost = Jev on all messages + Sonnet on the non-accepted share.

Calibration: accuracy within reported-confidence bands

Keyword check (check 3) behavior

Robustness and secondary checks

Sampling noise

With about 730 auto-accepted answers per model at the 70% setting, the standard error of each accuracy estimate is about 1.1 points (p ≈ 0.91). The 0.4-point difference between Jev and Sonnet is well inside it. We therefore describe the accuracy result as a tie, and the verdict rests on the pre-registered tolerance, not on Jev being ahead.

Unequal realized volume

At the 70% target, realized auto-accept rates were Jev 73.1%, Sonnet 75.3% and Haiku 81.4%, because Claude's confidences are coarse. As a check outside the pre-registered rule, we compared each model's 730 most confident answers that pass checks 1–3: Jev 91.4%, Sonnet 91.8%. The conclusion is unchanged.

Coarse confidence

Across all thresholds on the test set, Jev has 67 distinct achievable operating points, Sonnet 25, and Haiku 13. Haiku jumps from 0.8% to 45.7% auto-accepted between two adjacent settings.

Description tilt (contender F)

The v2 descriptions were tuned on Jev's errors and then given to Claude. To measure the resulting tilt, we ran Sonnet 5 on v1 descriptions (F). How F would be reported was committed to the changelog before it was run. Sonnet gained +2.4 / +0.9 / +0.8 points at the 50 / 70 / 85% rows from v1 to v2 (+1.9 on all answers), about the same as Jev's +0.9 at 70%. We found no measurable tilt toward Jev. Descriptions tuned on Sonnet's own errors were not tested.

Tuning vs test

On the tuning set, Jev led Sonnet by 3.6 points on all answers (83.0% vs 79.4%). On the test set, Sonnet was 0.7 points ahead (82.3% vs 81.6%). Part of the tuning-set lead was overfitting to the messages the descriptions were rewritten from, which is why thresholds and descriptions were fixed before the test set was opened.

Round 2: DeepSeek V4 Flash (contender J)

Pre-registered on 2026-10-02, before the full run. Model deepseek/deepseek-v4-flash via OpenRouter, provider pinned to streamlake/fp8 with no fallbacks, reasoning off, the same prompt as Claude. Confidence is the model's own probability for the category text it wrote (the product of that text's token probabilities, from log-probabilities) × 100. Its self-reported number was logged as a secondary score. Same sets, gate, keywords and threshold procedure; the win condition is the same 70% row, against Jev B.

Secondary: cleaned answer key

Pre-registered on 2026-10-13, before any cleaned score was computed. The official results above are unchanged.

  1. Candidates: the 206 test messages where at least 2 of the 4 v2 runs (Jev B, Sonnet C, Haiku D, DeepSeek J) picked the same category, different from the answer key. Ties between alternatives go to the alphabetically first.
  2. Blind review: each message shown with the key and the alternative as "label 1" and "label 2" in random order (seed 13), without saying which is official or which models picked what. Verdict: label 1, label 2, both fit, or unclear (unclear keeps the key).
  3. Who decided: Claude drafted every verdict; the author approved all 206 in bulk without overriding any. Outcome: key kept 55, other label 48, both fit 86, unclear 17.
  4. Scoring: thresholds unchanged. Cleaned accepts any accepted label (both, where both fit). Strict, added after the draft numbers were seen, treats "both fit" as the official key only, so only clear relabels count. It's a lower bound.

Jev minus DeepSeek, by answer key

Added 2026-10-14 after a reader's suggestion; exploratory, not pre-registered. Each model's n most confident answers that pass checks 1–3 (matched coverage). Ranges are paired 95% bootstrap intervals over the 1,000 test messages (2,000 resamples, seed 0) with each model's accepted set held fixed. They reflect which messages were sampled, not run-to-run model variation (one run per model).

Reading: fixing clear label errors leaves the gap essentially unchanged (6.2 → 5.9 points at Jev's 70% volume), because those errors cost both models about equally. Accepting either label where both fit narrows it (to 4.1 points, and about 2 points among the 500–600 most confident answers). The reason is confidence, not label choice: over all 86 both-fit messages the two models pick the alternative about equally often (Jev 64, DeepSeek 66), but DeepSeek is far more confident on them. Among each model's 500 most confident answers, 19 of DeepSeek's 39 original-key errors are both-fit messages, against 1 of Jev's 12. At Jev's 70% volume (731 answers), DeepSeek has 45 more original-key errors than Jev: 13 more on both-fit messages, 4 more on clearly mislabeled ones, and 28 more on messages nobody disputed (37 vs 9). The interval excludes zero on every key at every volume of 400 or more. Every reviewed label, with what each model said, is listed on the label audit page and in results/label_changes.csv (python label_audit.py).

Caveats: one reviewer (Claude, approved by the author); only messages where models agreed against the key were reviewed, which favors popular answers and can't catch errors no model disagreed with. The contenders' order is the same under all three keys. Files: review/, cleaned_scores.py, results/results_table_cleaned.csv.

Limitations

Reproducibility

pip install -r requirements.txt
python test_gate.py            # gate unit tests
python analyze.py              # recompute every table from the committed logs (no API calls)
python cleaned_scores.py       # secondary: re-score against the cleaned answer key
python build_site_data.py      # rebuild docs/data.json for this site

# re-run a model (costs money; needs OPENROUTER_API_KEY)
python run.py --model jev    --set tuning --categories v2
python run.py --model sonnet --set test   --categories v2 --confirm-test