Comparisons · Last updated 2026-09-21

Jev vs online experimentation

Online experimentation (A/B, interleaving) estimates user-outcome lift. Jev is not an experimentation platform. It can be a treatment: variant B uses a new question pack.

Unofficial. Flags still assign buckets. docs.typesafe.ai.

Comparison scope

If jev-latest moves mid-test, your experiment is confounded. Pin both variants’ model ids and git SHAs of criteria.

Criteria that decide the architecture

Decision quality and control

A better Noul calibration that tanks conversion is a failed ship. Optimize the action outcome, not model self-love.

Integration trade-offs

Flag → variant question module → same logging schema → experiment platform. Offline harness gates who is eligible to enter the A/B.

TypeSafe’s public models page lists jev-1.13 at $0.042 per million input tokens with output tokens free — a vendor claim, not a jev.pro measurement. Confirm on the models page before you bid.

When each approach fits

Prefer Jev when

• You want to test a decision policy

Prefer Online experimentation when

• You need the experimentation product itself

Offline first, online second, flags assign, Jev executes a variant.

What this page does not claim

• No invented lift.

• Schema-safe output is not the same as factual correctness.

• No independent bake-off numbers live here.

When not to use Jev

Do not use Jev as the stats engine. It will not assign units, compute p-values, or stop a test. Peeking with a Choice (“is variant B winning?”) is still peeking.

Do not treat a higher Noul or confidence as a ship gate. A/B floors are user outcomes. Jev floors are decision quality on labeled pairs. They are not interchangeable.

Do not run an experiment on jev-latest. Pin jev-1.13.0 (and the control’s id) for the life of the test. Criteria edits mid-flight are the same class of confound as changing the button copy.

Skip Jev when you are only testing copy, color, or price — unless the variant is a new question pack.

Pairing-specific failure modes

• Alias move. Marketing line jev-1.13 or jev-latest retargets while the experiment is live. The lift is then a model-release delta.
• Date windows as language. “Is this user in the holiday cohort?” is a timestamp compare in code, not a Noul.
• SRM from 429s. One variant calls Jev more often and sheds traffic on rate limits. Guardrail the error budget, not only conversion.
• Schema-safe winner. A well-typed ship_b key from Jev is not a sequential test. Use your stats stack.

Production sketch

1. Offline harness on pinned jev-1.13.0 vs control questions (offline evaluation). Fail closed if labeled floors miss.
2. Feature flag assigns units (feature flags). Both arms log the same schema: unit id, variant, response.model, criteria SHA, action taken.
3. Jev executes the variant’s questions only. Code applies that arm’s τ. Do not let Jev pick the arm.

# Compose rule: the flag is the assignment. Jev must not peek the metric.
if flag.bucket(unit) == "b":
    ans = jev_eval(state, QUESTIONS_B, model="jev-1.13.0")
    act = act_from(ans, tau_b)
else:
    act = control_policy(state)
experiment.log(unit, flag.bucket(unit), act)  # stats platform reads this

4. Primary metric stays the product outcome. Guardrails: review rate, 429s, latency.
5. Freeze questions until the test ends. Then promote by SHA, not by vibe.

Hub: Comparisons. Siblings: vs feature flags, vs offline evaluation.

FAQ

Can Jev pick the winning variant? That’s peeking. Use your stats stack.

Multi-armed bandits? Your code. Jev is not a bandit API.

Are TypeSafe cookbook lifts an A/B? No. They are vendor-measured evals on their pages. Your experiment is your users.

Disclaimer

This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.