Jev evaluation guide
Evaluation means: did the product action match gold, at a cost you accept? TypeSafe’s workflow charts are a vendor story (including high-end speed multiples). Your harness is the review.
Unofficial methodology — not a review blog and not a jev.pro leaderboard. How-to depth: offline evaluation. Canonical APIs: docs.typesafe.ai. Replicas are not Jev and not endorsed.
Independent angle (cover, do not clone)
Win delta (review-hands-on): reproducible eval notes with dates; methodology > opinion blog.
Harness recap
Gold JSONL → pin jev-1.13.0 → freeze questions → score label and act/review/abstain → CI on diffs. Cookbook lifts stay on cookbook pages.
Dated methodology (2026-09-21)
Record door, language mix, whether state was filtered, and who labeled gold. Author-bias exists (TypeSafe even warns their evals were written by the capabilities team). Yours will too — write it down.
What “good” means
Good is policy: fewer expensive false autos, acceptable review load, no silent 422s. A higher argmax match rate with more irreversible mistakes is a regression.
Date every report. Include door + pin + question SHA. If you change two of those at once, you cannot attribute.
Policy-good, not argmax-good
Score auto / review / abstain / block against gold, not only whether the top Choice string matched. Noul 0.5 is officially yes/no equally likely — if your mapping autos there, the harness should fail the policy, even if the label “yes” was the mode. Do not reuse a Noul floor on a Choice. Do not treat Score expectation as a dollar or a minute.
TypeSafe’s workflow charts and homepage multiples (they have cited figures such as 193.6× faster) are a vendor story, including their own caveats about author bias and wrapper baselines. Your harness is the review. Testing twin: testing guide. Comparison of offline shapes: vs offline evaluation.
Date the door, the pin, and the SHA
Every report header should be boring:
date: 2026-09-21
door: first-party HTTP
model: jev-1.13.0
question_sha: <git>
language_mix: ...
state: filtered | raw
gold_authors: ...
metrics: auto_precision, review_rate, 422_countIf you change two of door / pin / SHA at once, you cannot attribute. jev-latest can move; pin if floors exist. Author-bias exists (TypeSafe even warns their evals were written by the capabilities team). Yours will too — write it down.
Online eval comes after offline gates. Public TypeSafe materials only — public evals. We will not average blog charts into an unofficial IQ.
When not to use this page’s advice
Do not invent a 193× tile on an internal dashboard. Do not score SemIf/OpenJev and title the doc “Jev eval.” Do not claim schema-safe answers are factually correct. Do not skip humans on the gold set for high-risk actions.
What this page does not claim
- No invented 193× on jev.pro dashboards.
FAQ
Online eval? After offline gates. See vs online experimentation.
Public leaderboards? public evals if we list any — still not Jev-as-a-score.
Is argmax accuracy enough for launch? No. Score the policy action. A clever top-1 with a bad auto is a fail.
May I mix doors in one report? Only if you split rows. Door strings and errors are not interchangeable.
Disclaimer
This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.
Open-cluster pages are independent field-guide notes. Replicas and third-party interfaces mentioned anywhere on jev.pro are not Jev and not endorsed. Hub: Open. Siblings: testing guide, offline evaluation, vs offline evaluation. Canonical: https://docs.typesafe.ai.