jev.proField guide · curious → Pro

Open · Last updated

Jev testing guide

Testing Jev is not screenshotting Playground. It is frozen questions + pinned model + gold cases + assertions on actions. Date: treat this page as 2026-09-21 methodology.

Independent test methodology — not a hands-on review blog and not TypeSafe QA. Pair with evaluation guide. Canonical: docs.typesafe.ai. Replicas are not Jev and not endorsed. We do not issue keys.

Independent angle (cover, do not clone)

Win delta (review-hands-on): reproducible eval/test notes with dates; methodology > opinion blog. Not a clone of a “Jev review” URL.

Layers

  1. Contract tests — request JSON schema, door model id, no boolean on first-party.
  2. Unit tests of your policy — given stubbed answers, action enum is correct.
  3. Golden network tests — small recorded set against pinned jev-1.13.0 (flakes possible; quarantine, don’t ignore).
  4. Adversarial cases — injection, contradiction, huge state (expect degradation).

What not to assert

Bit-identical probabilities forever. Exact vendor latency. “Cannot be wrong.”

Dated lab note template

date: 2026-09-21
model: jev-1.13.0
door: first-party HTTP
question_sha: <git>
n: 80 gold rows
metrics: auto_precision, review_rate, 422_count
notes: English tickets only; no images

Fill numbers from your run. Do not paste ours — we did not run your set.

Flake policy

Golden calls against a hosted model will flake (429, 529, rare answer flips). Cap retries. Quarantine known-soft cases. Never “sleep and hope” in CI.

Adversarial fixtures belong in the repo: injection, date-math bait, hex colors vs names (jaggedness). Expect the model to be weak where TypeSafe said it is weak.

Flake classes to name in CI

Name the classes instead of a blanket “retry”:

Class Treat as Do not
429 / 529 Retry cap, then quarantine the job Sleep-and-hope loops
401 Infra fail — missing TYPESAFE_API_KEY “Flaky model”
422 Pack bug — fail the SHA Retry the same body
Rare answer flip Soft-quarantine with a ticket Assert bit-identical probabilities forever
Timeout Bound SDK retries (official: 429/529 only) Unbounded backoff in CI

Nightly job holds the secret; PRs mock the client. Pin jev-1.13.0 on goldens. jev-latest makes the suite a moving target. Date every lab note. Fill numbers from your run — we did not run your set.

Do not assert exact vendor latency. Do not assert “cannot be wrong.” Schema-safe ≠ factual correctness. Score policy actions, same as evaluation.

Adversarial fixtures belong in the repo

Expect jev-1.13 to be weak where TypeSafe said it is weak. Check in:

These are not “mean tests.” They are the production remainder. Pair with best practices and offline evaluation.

Do not run the same fixtures against SemIf/OpenJev and file a “Jev flake.” Different weights, not Jev and not endorsed.

When not to use this page’s advice

Do not screenshot Playground and call it coverage. Do not publish a pretended jev.pro leaderboard. Do not disable flakes by deleting the gold row. Do not load-test the vendor door from CI because retries were left on.

What this page does not claim

FAQ

CI without keys? Mock the client in PRs; nightly job with a secret for goldens.

SDK retries in tests? Disable or bound them or your suite becomes a load test.

Must probabilities match bit-for-bit across days? No. Assert actions and bands. Quarantine known-soft cases.

Can I test on jev-latest to “stay current”? Only in an isolated job. Floors belong on a pin.

Disclaimer

This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.

Open-cluster pages are independent field-guide notes. Replicas and third-party interfaces mentioned anywhere on jev.pro are not Jev and not endorsed. Hub: Open. Siblings: evaluation guide, offline evaluation, best practices. Canonical: https://docs.typesafe.ai.

Sources