Measuring decision quality with Jev
Decision quality for Jev is whether your code took the right action, not whether the argmax matched a blog chart. Jev is TypeSafe’s first public System One model: typed probabilistic decisions, no text generation. TypeSafe publishes workflow evals and jaggedness notes. Those are vendor materials. Your harness uses your labels.
This page is for builders who already heard of Jev / System One and want a production-minded mental model. jev.pro is an independent unofficial field guide — not affiliated with TypeSafe AI. Confirm facts on docs.typesafe.ai. We do not sell keys.
Definition
Quality is a property of the loop: state you assembled, atomic questions you asked, thresholds you applied, action you took. A perfectly schema-safe Choice can still route the ticket to the wrong queue. Schema-safe ≠ correct. “Can’t hallucinate” does not mean the action was right.
Do not paste homepage multiples such as 193.6× into a dashboard as if you measured them. TypeSafe itself says those workflow-eval figures sit on the higher end of real-world gains. Label vendor claims.
Mental model: four scores, never one site-wide accuracy
- Action agreement with gold (auto-allow / hold / route), including abstentions.
- Calibration of confidence or noul on held-out traffic. If the model says 0.8 often, about 80% of those cases should match gold. That is a group statement — one miss is allowed.
- Review rate vs auto precision — a cheap model that holds everything is not “high quality.”
- Operational: 422 rate, 429 rate, token p95,
response.modelmix.
Plot per question id. Never a single site-wide “AI accuracy.” RLCD is the named training objective for calibration; you still measure it on your distribution.
How to run the harness
Freeze questions and pin jev-1.13.0 (or the current versioned id). Store traces: state hash, question module version, full probabilities, action taken. Replay the same POST — there is no hidden replay endpoint you should invent. Compare actions, not only argmax, when you promote. See deterministic outputs.
Keep arithmetic out of the model. Jaggedness for 1.13: weak at numeric precision, extra indirection, literal wording. If gold needs a sum, compute the sum in code and ask Jev about the remainder.
English is the primary language. If you evaluate other languages, do not assume a Noul of 0.9 means the same thing until you measure it (language support).
What not to score
- Invented public leaderboards or search-volume ranks. We do not invent those.
- Vendor cookbook lifts copied as your lift. Cite them as vendor, on their pages.
- Averaging unrelated Nouls into one “quality” number unless you validated that hack.
- Treating TypeSafe’s 70ms–500ms or $0.042/M input as a quality metric. Those are cost/latency claims, not correctness.
Quality is not an API field. You compute it.
See also
FAQ
Can I average Nouls?
Only as a hack you validate. Official: questions are different judgments.
Vendor cookbook lifts?
Cite them as vendor, on their pages.
Is quality an API field?
No. You compute it.
Does pinning jev-1.13.0 freeze quality?
It freezes the weights behind the alias. Change criteria or state filters and you need a re-score. Schema-safe ≠ correct.
Disclaimer
This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.