Glossary: Calibration
Calibration means a reported probability (or confidence) matches observed frequency: events with p ≈ 0.8 happen about 80% of the time across a group. It is not a warranty that the next ticket’s winner is true. TypeSafe’s launch post says Jev is trained with RLCD for honest probabilities and that higher confidence means higher accuracy. That is a vendor claim. You still measure it on your labels.
This glossary entry is for builders who already heard of Jev / System One and need a citable definition before they wire thresholds. jev.pro is an independent unofficial field guide — not affiliated with TypeSafe AI. Confirm field names against docs.typesafe.ai. We do not sell or issue API keys.
How it shows up
On POST /v1/systemone, Choice and Score return probabilities plus a derived confidence. Noul returns only noul in [0, 1] — no confidence field — so you calibrate the noul itself. Measure ECE or reliability diagrams on your labels. Do not treat launch-post plots as your ECE.
Pin the versioned model id before you compare weeks. jev-latest currently aliases jev-1.13.0 on the vendor snapshot we read; aliases move. Official System One language is careful: calibration does not guarantee that an individual answer is correct. Use confidence as a gate in code, not as a per-row certificate.
TypeSafe’s jaggedness note for jev-1.13 also says the model is weak at numerical calibration of Score expectations as physical quantities. Threshold “past level 1”; do not reconstruct a dollar amount from a fractional score.
Contrast
Calibration vs confidence. Confidence is one number derived from the shape of a Choice/Score distribution. Calibration is whether those numbers (or the raw probabilities) line up with hit rates. You can have a peaked wrong answer: high confidence, poor calibration on that slice.
Calibration vs RLCD. RLCD is TypeSafe’s name for the training method. Public docs name it; they do not publish a full paper. Do not invent a loss. Judge calibration by labeled traffic.
Calibration vs a confidence threshold. A threshold is a cut you chose (TypeSafe examples often show 0.5 as “unsure” and 0.9 for high-stakes acts — illustrations, not defaults). Fitting a cut is not the same as proving the bins are reliable.
Calibration vs schema safety. A 200 response with a closed choice is schema-safe. Schema-safe ≠ factually correct. Calibration talks about frequencies, not about a single valid key being the right department.
Common mistakes
- Averaging Nouls across different questions as if they were one probability. Official: questions are different judgments. Only average as a hack you validate.
- Comparing this week’s ECE to last week’s while
modelwasjev-latestand the alias moved. - Copying a Noul τ onto Choice confidence. Different statistics.
- Treating TypeSafe’s 0.5 / 0.9 examples as shipped defaults.
- Asking Jev to “apply the calibration formula.” Keep math in code.
See also
FAQ
Can I average Nouls across questions?
Only as a hack you validate. Official: questions are different judgments.
Does pinning the model help?
Yes. Aliases move. Compare calibration against a versioned id such as jev-1.13.0.
Is calibration a response field?
No. You compute it offline from stored answers and labels. Persist the full probabilities map in a decision trace.
Does high confidence mean this row is correct?
No. Group-level claim only. Gate irreversible actions harder.
Limits
This is a jev.pro working definition unless TypeSafe uses the same word as an API field. We do not invent search volume. Schema-safe answers are not automatically factually correct.
Disclaimer
This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.