Security evaluation with Jev
SIEM, EDR, and SOAR already detect and can isolate. Jev judges whether the alert narrative + evidence pack is enough and whether policy allows the proposed contain class. Code aborts auto-containment unless those atoms clear. Jev does not match IOCs, isolate hosts, or write detections.
This unofficial page is the evaluation slice of the security containment-abort pack. Intent: apply the Jev (TypeSafe System One) decision model to security containment-abort evaluation. Primary search language: Security Jev evaluation. Confirm patterns on docs.typesafe.ai. This site does not sell, issue, or proxy TypeSafe keys. Use a credential you already have from the console or a documented gateway.
Independent angle (cover ≠ clone): Evidence-sufficiency + policy-allow + abort thresholds before SOAR fires isolate/disable — not a clone of support-action recipes, LLM guardrail Noul screens, or SRE incident SEV classification.
Security use-case context
Evaluation for security containment-abort is a frozen harness, not a vibe check and not an opinion-blog “Jev review.” Labels: abort-reason gold from analysts, evidence_ok gold, and whether isolate was justified. We publish no unofficial accuracy.
Hub: Use cases. Compare, when the other tool is the real job: security automation. Broader eval notes: evaluation guide.
Evaluation inputs
Replay the same contract you ship:
{
"alert": { "id": "SOC-1902", "text": "Impossible travel + new OAuth grant on finance-sso; user says travel is unexpected." },
"asset": { "tier": "prod-idp", "owner": "identity" },
"evidence": { "ioc_hits": 0, "enrichment_ok": true, "summary": "No hash match. New grant to unknown app. Last login US-East 40m earlier." },
"policy": { "isolate": "Auto-isolate only if evidence_ok and asset.tier is not shared-idp.", "evidence_min": "Need either IOC hit or named user contradiction plus enrichment_ok." }
}Freeze questions, criteria, and jev-1.13.0. Record the response model. Pair each row with ioc_hits and asset.tier computed outside Jev — the harness must score the conjunction your abort gate actually ran.
Decision signals and actions
Score these, not a blog-grade star rating:
- False isolate on a planted thin-evidence canary
- False-abort rate (analyst load / missed contain)
- Disagreement after a policy-text edit
- Adversarial “ignore criteria” alert text
contain_class == otherrate (coverage hole, not a model win)
Pair auto-act errors with handoff rate. If the abort gate never acts, you have not evaluated security containment-abort — you have evaluated a human queue. The distinctive metric is false isolate when evidence_ok should have aborted.
Do not treat a Noul of 0.5 as a “medium” security containment-abort score — it means yes and no are equally likely. Conjunctions stay in your code.
Guardrails and escalation
Promote a threshold only when the harness says auto-isolate error ≤ SLA and analysts still catch the residual. TypeSafe’s confidence-gated examples use a lower bar for recoverable reads than for irreversible actions. Those numbers are illustrations. For security containment-abort, treat auto_isolate_host as the high bar (isolating a host or disabling a credential). Tune on labels — see offline evaluation.
Evaluation and rollout notes
After any policy or criteria edit, rerun before production. Cookbook lifts you see on TypeSafe pages are vendor claims — re-measure on your alert + evidence packs. Pin jev-1.13.0 (the versioned id) after you fit thresholds. jev-latest and the marketing line jev-1.13 can move. Log the response model. TypeSafe’s published list price for jev-1.13 is $0.042 per million input tokens (vendor claim — confirm on the models page); output tokens are free on that same page. Unused distractors still bill as input.
Official Python and JavaScript SDKs read TYPESAFE_API_KEY and retry documented 429/529. This site does not sell, issue, or proxy TypeSafe keys. Use a credential you already have from the console or a documented gateway.
Pack map
| Slice | Page |
|---|---|
| Graph and primitives | decision workflow |
What may enter state |
input contracts |
| What to gather first | evidence collection |
| Atomic rules | policy checks |
| Act / review / abstain | confidence thresholds |
| Reviewer payload | human handoff |
| What to persist | audit trail |
| How it breaks | failure modes |
| Labeled replay | you are here |
| Shadow → canary | production rollout |
FAQ
Will jev.pro publish a leaderboard for this use case? No. Measure on your labels. Vendor cookbook figures stay labeled as vendor claims.
What must stay frozen? Questions, criteria, and the pinned model id. Aliases can move. Policy SHA is part of the gold.
Where is the rest of the Security pack? Start with Security failure modes and Security production rollout. Cluster hub: Use cases.
Is this the LLM guardrails cookbook? No. Guardrails screen prompts and completions. This pack aborts containment when evidence or policy is thin. See LLM guardrails.
If IOC hits are already ≥ 1, should we wait for Jev? No. Deterministic contain wins. Jev is for leftover messy narrative.
What this page does not claim
- Not a SIEM, SOAR, EDR, or certified safety filter.
- No MTTD / MTTC benchmarks.
- Not official TypeSafe.
- Official TypeSafe status, or that jev.pro issues API keys.
- That a schema-constrained answer is automatically factually correct.
Disclaimer
This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai.
Primary documentation: https://docs.typesafe.ai. Hub: Use cases.
Sources
Public TypeSafe or adjacent documentation only. No private claims.