Questions for analysts
Analysts should refuse dashboards that plot a TypeSafe blog multiple as our KPI. Ask for action-level metrics on pinned models. This brief is unofficial and independent.
This page is for builders crossing from curious to production-minded Jev use. jev.pro is an independent unofficial field guide — not affiliated with TypeSafe AI. Cover intents; do not clone rival IA. Confirm facts on docs.typesafe.ai.
Open-cluster pages are independent field-guide notes. Replicas and third-party interfaces mentioned anywhere on jev.pro are not Jev and not endorsed.
Ask
- What is the unit — label or action?
- Which model id?
- How was gold labeled?
- Review rate vs auto precision tradeoff?
- Token p95 and 429s?
- Split by language?
- Did we reuse a Noul threshold on a Choice?
- Are Score expectations abused as magnitudes?
Plot these first
Auto precision, review rate, 422 rate, token p95, response.model mix. Then, if you must, calibration curves per question id. Never a single site-wide “AI accuracy.”
Gold sets before charts
Refuse a dashboard that has no gold. Ask:
- Who labeled, on what date, in which language mix?
- Was gold an action (
auto/review/block) or only an argmax label? - Did labelers see the same filtered
statethe door saw? - Is the set frozen against a question SHA, or do people quietly edit criteria after the fact?
TypeSafe even warns that their workflow evals were authored by the capabilities team — author bias exists. Yours will too. Write it down. Methodology lives on the evaluation guide; warehouse hygiene lives on for data teams.
A gold set of 30 “easy English tickets” will make every pin look brilliant. Ask for the ugly remainder: injection, date bait, mixed language, missing fields.
Calibration is per question, not per “AI”
Official: questions are evaluated independently on the same state. A Noul on policy_ok is not interchangeable with a Choice confidence on intent. Plot reliability per question id, on your labels. TypeSafe trains for calibrated decisions — treat that as a vendor claim until you see a reliability curve on your traffic.
Noul 0.5 means yes and no equally likely. If a product floor sits at 0.5 for an irreversible action, the chart should scream. Do not reuse a Noul threshold on a Choice. Do not treat Score expectation as a magnitude (dollars, minutes, counts) — jaggedness: do not reconstruct numbers from Score levels.
Dashboards that must not ship
- A tile that reprints a TypeSafe launch-post multiple (they have cited high-end speed/cost figures) as our KPI.
- One site-wide “AI accuracy” that averages Nouls with Choice confidences.
- Token cost computed from list price $0.042 / M input (vendor; output free per models page — confirm) without measured
usage.input_tokens. - A mix of
response.modelvalues hidden behind one pin name.
Plot first: auto precision, review rate, 422 rate, token p95, response.model mix, 429/529 counts. Then calibration. Then, if leadership wants a speed story, link the vendor post and show your p95 next to it.
When not to use this page’s advice
Do not use these questions to “prove Jev” against an open replica. SemIf/OpenJev results are not Jev and not endorsed. Do not put vendor cookbook lifts into a board deck without the vendor label. Do not ask Jev to compute the dashboard — aggregates and date diffs are SQL.
See also
FAQ
Can I average Nouls across questions?
Only as a hack you validate. Official: questions are different.
Vendor cookbook lifts?
Cite them as vendor, on their pages.
Do we have a canned ECE?
No. Compute it on your labels.
May I paste 193× into Grafana?
No. That is a TypeSafe workflow-eval claim. Link their post; show your p95.
Disclaimer
This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.