autotrust/JEV — an open-weights student of TypeSafe Jev 1.13 (Qwen3.5-9B, calibrated System One decisions)

Two different models, two different organisations — please read the names carefully. TypeSafe Jev 1.13 is the hosted, closed model made by TypeSafe AI; it is the teacher whose public outputs this model was trained on. autotrust/JEV (this repository) is an independent open-weights student built by autotrust from the Apache-2.0 corpus SargeDev/jev-distill-corpus-v3. autotrust/JEV is not affiliated with, endorsed by, or a product of TypeSafe AI. Throughout this card the teacher is always written TypeSafe Jev 1.13 and this model is always written autotrust/JEV.

One forward pass in, a calibrated probability distribution out. autotrust/JEV answers typed questions (noul yes/no · choice over 2–16 options · score on a 0–5 scale) about any text or JSON state. No JSON parsing, no prompt engineering — it was distilled from the full output distributions of TypeSafe Jev 1.13 onto a pretrained Qwen3.5-9B backbone. The bundle ships two heads on one backbone: the 24-slot decision head (typed decisions) and the original lm_head (ordinary text generation). The backbone weights on disk are bit-identical to Qwen3.5-9B; the decision adapter is a separate 154 MB LoRA that is only applied for decisions — so generation quality is exactly the base model's.

Model family — same recipe, same API, same two-head packaging: autotrust/JEV (this repository, Qwen3.5-9B, 18 GB) · autotrust/JEV-27B (Qwen3.8-27B, 54 GB: KL 0.019, choice 90.3 %, out-of-distribution KL 0.104, HumanEval 78.0 %).

Headline results

Held-out test set: 29,955 questions, 53 domains, all three question types. Numbers are for autotrust/JEV.

What is measured	autotrust/JEV	How to read it
How closely it reproduces the teacher's probabilities — mean KL divergence, lower is better, 0 = identical	0.021	The student's probability distributions are nearly indistinguishable from TypeSafe Jev 1.13's
Yes/no questions (noul) — AUROC, higher is better	0.996	Ranks true vs. false almost perfectly
Multiple-choice questions (choice) — picks the same option as the teacher	89.8 %	Over all questions, including those where the teacher itself is nearly tied
Multiple-choice questions where the teacher has a clear favourite (its top two options differ by ≥ 0.1)	95.4 %	On the remaining, near-tie questions any faithful copy would agree only about half the time
Rating questions (score, 0–5 scale) — mean error of the expected rating, lower is better	0.10	About one tenth of a rating step
Calibration — expected calibration error, lower is better	0.0007	A stated 80 % is an 80 %; no post-hoc temperature correction was needed (fitted temperatures ≈ 1.00)
Tasks never seen in training (Open-Jev OOD split) — mean KL divergence	0.234	Still tracks the teacher closely on unfamiliar task families
Independent benchmark with human labels (intent / topic / emotion classification, not in our training data) — accuracy relative to TypeSafe Jev 1.13	97 % with 2 options, 90 % with 16	Degrades faster than the teacher as the option list grows; autotrust/JEV-27B keeps 96 % at 16. See Tested on fresh, real-world data
Text generation (AR head) — HumanEval pass@1, greedy	70.7 %	Identical to Qwen3.5-9B (116/164, byte-identical completions): the generation path is untouched
Latency — one request on one B200 GPU, median	≈ 90 ms	2.5 ms per decision when requests are batched

Only 40.2 M parameters (0.5 % of the 8 B backbone) were trained, for ≈ 3 GPU-hours on a single B200 (0.93 epoch ≈ 608 k rows). Full evaluation, including robustness and per-domain breakdowns, is in Evaluation details and in code/reports/eval_s2_9b_epoch1.md.

Runs on vLLM — one engine serves both heads from the same weights through an OpenAI-compatible API; text generation is ≈ 50× faster than the PyTorch path. See Inference with vLLM.

Tested on fresh, real-world data

Two checks on inputs the model has never seen: an independent benchmark with human gold labels on which TypeSafe Jev 1.13's own scores are public, and a set of decisions on Hacker News and V2EX content from 23–25 September 2026. The 27 B sibling is shown for comparison.

Independent benchmark: side by side with TypeSafe Jev 1.13

gazelle93/decision-models-under-pressure (published 25 Sep 2026) asks decision models to pick the right label for real texts from CLINC-150, MTOP, GoEmotions, DBpedia and financial tweets, under three kinds of pressure: more options, near-miss options, and shuffled option order. The labels are human gold labels, none of this data is in our training set, and TypeSafe Jev 1.13's results are published with it. We re-ran the same items, up to our 16-option limit.

	TypeSafe Jev 1.13 (published)	autotrust/JEV (9 B)	autotrust/JEV-27B
Accuracy with 2 / 4 / 8 / 16 options (800 items, 4 domains)	0.890 / 0.801 / 0.782 / 0.769	0.868 / 0.774 / 0.735 / 0.694	0.876 / 0.784 / 0.767 / 0.740
16 options — CLINC / DBpedia / GoEmotions / MTOP	0.945 / 0.900 / 0.470 / 0.760	0.875 / 0.855 / 0.325 / 0.720	0.930 / 0.885 / 0.415 / 0.730
16 options, near-miss vs. unrelated wrong options (CLINC + MTOP, 400 items)	0.912 vs 0.985	0.875 vs 0.975	0.907 vs 0.983
Answers changed by shuffling the options alone (16 options, 5 orderings)	7.0 %	11.5 %	7.4 %
Time for 14,400 decisions on one B200	—	42 s	110 s

With few options the 9 B student is within 2–3 points of the teacher on data it was never trained on (97 % of its accuracy at 2 options). As the list grows it falls behind faster: 90 % of the teacher's accuracy at 16 options, against 96 % for the 27 B sibling. It is also more sensitive to option order (11.5 % of answers change vs 7.0 % for the teacher) and loses a little more on near-miss options. It is 2.6× faster than the 27 B model. Our run follows the benchmark's published method (gold plus the first K−1 distractors of a pool, shuffled per item); the orderings are seeded differently, so compare the aggregates, not individual items.

Fresh examples (Hacker News and V2EX, 23–25 September 2026)

The expected answers were written by hand before the model was run. These are illustrations (≈ 110 decisions), not a benchmark.

task	autotrust/JEV (9 B)	autotrust/JEV-27B
Topic of 19 HN front-page stories (10 options) + "is it about AI?"	38 / 38	38 / 38
12 comments from a heated HN thread: "insults or attacks someone?" + "what is it mainly doing?" (6 options)	22 / 24	23 / 24
10 V2EX hot posts in Chinese: "contains a referral / invite code?" + "promotes a product or paid offer?"	18 / 19	19 / 19
Community use cases: code-rule checks in the style of adhere, injection filtering, ticket routing, phishing, code-review diffs, urgency scores	14 / 15	15 / 15

input	question	autotrust/JEV (9 B)
HN: "Two-tier encryption in the UK"	topic (10 options)	security & privacy · 0.87
HN: "Using LLMs to trace alchemical knowledge and decode 17th century letters"	about AI?	P(true) = 0.88
HN comment: "Why do you talk like that? Are you one of the redditors that has been invading…"	insults or attacks someone?	P(true) = 0.67
HN comment: "Please stop this. We've asked you before to observe the guidelines…"	what is it mainly doing?	moderating the discussion · 0.78
V2EX: "一个不需要 gemini pro 的完全免费的注册 Muse 的方法 … 邀请码：…"	contains a referral / invite code?	P(true) = 1.00
V2EX: "今天中秋节，还要加班的有吗？来报道下"	promotes a product or paid offer?	P(true) = 0.01
Diff replacing a parameterised query with "… WHERE id = " + request.args["id"]	introduces a security vulnerability?	P(true) = 0.93 (0.17 for a variable rename)
"I'm not happy with the fit. What are my options here?"	asking for a refund?	P(true) = 0.17 (TypeSafe's docs report 0.22 for Jev on this exact text)

Where it failed or wavered — and where the 27 B sibling did better:

• Code-rule check: it missed a TypeScript file that declares const port: number = Number(process.env.PORT) against the rule "a port must be a branded, range-checked integer" (0.33); autotrust/JEV-27B flags it (0.93).
• Fraud screening: a CEO wire-transfer (business-email-compromise) message is flagged, but only at 0.56 (27 B: 0.84).
• Comment intent: "Because it's not a real argument. It's a deflection people use." read as attacking another commenter (0.66) rather than arguing a point.
• A poker spot with the nuts (a solver checks 100 %): shoves with 0.70; TypeSafe Jev 1.13 shoved with 0.62 in a published test, and the 27 B sibling with 0.63, so this mistake comes from the teacher.

Which size to pick: for routing, moderation, topic triage and short option lists the 9 B model gives nearly the same answers 2.6× faster. For long option lists (more than about 8), code-rule checks and fraud screening, use autotrust/JEV-27B.

Reproduce: code/scripts/realworld_examples.py (the header lists how to download the benchmark files; the HN and V2EX inputs came from their public APIs on 25 September 2026).

Inference with vLLM (recommended)

One vLLM engine serves both heads from the same pristine weights. Ordinary requests go through the base lm_head (text generation, exactly Qwen3.5-9B); requests addressed to the LoRA module jev-decision go through the decision head. adapter_vllm/ contains the backbone LoRA plus the 24-slot decision head re-expressed as an lm_head LoRA (only the 24 verbalizer rows change), so a typed decision is a single prefill step with max_tokens=1, constrained to the option tokens and read back as log-probabilities.

1 — Start the server (OpenAI-compatible)

hf download autotrust/JEV --local-dir JEV          # ~18 GB
vllm serve JEV --served-model-name autotrust/JEV \
    --enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV/adapter_vllm \
    --logprobs-mode processed_logprobs --max-model-len 4096

--logprobs-mode processed_logprobs is required: it makes the returned log-probabilities respect allowed_token_ids. Add --enable-prefix-caching --mamba-cache-mode align if you ask many questions about the same state (see notes below).

2 — Text generation: the unmodified base model

curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "autotrust/JEV",
  "messages": [{"role": "user", "content": "In one sentence, what is safety stock?"}],
  "max_tokens": 60, "chat_template_kwargs": {"enable_thinking": false}}'

3 — Typed decisions: the decision head (Python, only requests + two small JSON files)

import json, math, requests
from huggingface_hub import hf_hub_download

REPO, URL = "autotrust/JEV", "http://localhost:8000"
dh = json.load(open(hf_hub_download(REPO, "adapter_vllm/decision_head.json")))  # bias + verbalizer token ids
T  = json.load(open(hf_hub_download(REPO, "calibration.json")))["per_kind"]       # per-kind temperatures

def decide(kind, state, question, options=None):
    options = {"noul": ["false", "true"], "score": [str(i) for i in range(6)]}.get(kind, options)
    lines = options if kind != "choice" else [f"{'ABCDEFGHIJKLMNOP'[i]}) {o}" for i, o in enumerate(options)]
    prompt = f"[kind] {kind}\n[state] {state}\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"
    s = dh["slots"]["ranges"][kind][0]
    ids = dh["verbalizer_ids"][s : s + len(options)]              # the option tokens of this kind
    r = requests.post(f"{URL}/v1/completions", json={
        "model": "jev-decision", "prompt": prompt, "max_tokens": 1, "temperature": 1.0,
        "logprobs": len(options), "allowed_token_ids": ids,
        "add_special_tokens": False, "return_tokens_as_token_ids": True}).json()
    lp = {int(k.split(":")[1]): v for k, v in r["choices"][0]["logprobs"]["top_logprobs"][0].items()}
    z = [(lp.get(t, -1e9) + dh["bias"][s + i]) / T[kind] for i, t in enumerate(ids)]   # + head bias, / temperature
    e = [math.exp(x - max(z)) for x in z]
    return {o: x / sum(e) for o, x in zip(options, e)}

print(decide("choice", "SKU AX-330 stock at 8% of safety level; supplier late twice this quarter.",
             "Supplier response for this scenario.", ["issue_warning", "renegotiate", "dual_source", "maintain"]))
# ≈ {'issue_warning': 0.33, 'renegotiate': 0.14, 'dual_source': 0.53, 'maintain': 0.001}
print(decide("noul", "Customer says the parcel arrived damaged and wants their money back.",
             "Is the customer asking for a refund?"))

Values can differ in the third decimal between runs: vLLM computes in bf16 and results depend slightly on which requests are batched together. The adding of the head bias and the division by the temperature happen client-side; the log-softmax normaliser that vLLM applies cancels out, so the result is exactly the decision head's calibrated distribution.

Offline / batch (Python API)

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM("JEV", enable_lora=True, max_lora_rank=32, logprobs_mode="processed_logprobs", max_model_len=4096)
decision = LoRARequest("jev-decision", 1, "JEV/adapter_vllm")

gen = llm.generate(["..."], SamplingParams(temperature=0.0, max_tokens=256))            # base model, no LoRA
dec = llm.generate([prompt], [SamplingParams(max_tokens=1, temperature=1.0,             # decision head
                                              allowed_token_ids=ids, logprobs=len(ids))],
                   lora_request=decision)                                               # then + bias, / T as above

Mixed batches work too: pass a per-request lora_request list (None for generation, decision for decisions) and both heads are served in the same generate call. code/src/jev_judge/vllm_engine.py wraps all of this (VllmJudge(...).decide(rows), .generate(prompts), .mixed(...)).

Measured on one B200

workload	PyTorch path	vLLM
Text generation — 164 HumanEval completions (greedy, ≤ 384 new tokens)	165 s	3.3 s (≈ 50×)
Typed decisions, offline batch — 29,955 test questions	75 s (398 q/s)	80 s (374 q/s)
Typed decisions over HTTP — 64 / 256 concurrent clients	—	150 / 205 req/s
Decision fidelity vs. the PyTorch path	—	test KL 0.0211 (identical); mean |Δp| 0.0008 over HTTP
Many questions about one state, --enable-prefix-caching	—	+14–20 % throughput

Notes:

• The big win is on the generation head. Decisions are a single prefill pass with no decoding, so offline batch throughput is about the same as the PyTorch path; for decisions vLLM mainly buys serving: continuous batching under concurrency, an OpenAI-compatible API, and one engine for both heads.
• Prefix caching: this architecture mixes Gated DeltaNet and attention layers, and vLLM caches it in blocks of 528 tokens, so only shared prefixes longer than 528 tokens are reused. The template puts [kind] before [state], so only questions of the same kind share a prefix. On 293 real states × 7.7 yes/no questions each (≈ 480-token states) prefix caching served 19.6 % of prompt tokens from cache (+14–20 % throughput) with identical outputs.
• Requires a vLLM build with Qwen3.5 (qwen3_5) support, LoRA on lm_head, --logprobs-mode and allowed_token_ids; tested with a vLLM development build from September 2026. Start-up takes 3–8 minutes (CUDA-graph capture with LoRA enabled).

What it does

kind	question	returns
noul	"Is this statement true?"	[P(false), P(true)]
choice	"Which of these 2–16 options?"	one probability per option, aligned with your options
score	"Where on this ordered 0–5 scale?"	a distribution over the six levels (+ expected score)

[kind] choice
[state] SKU AX-330 stock at 8% of safety level; supplier late twice this quarter.
[question] Supplier response for this scenario.
[options]
A) issue_warning
B) renegotiate
C) dual_source
D) maintain
[decision]:

The template is tokenised as one string; the last token's final-norm hidden state goes through a linear fp32 head H → 24 slots (noul → slots 0–1, score → 2–7, choice → 8–23). Inactive slots are masked, a per-kind temperature is applied, and a softmax yields the distribution aligned with your options. One prefill pass — no decoding.

One backbone, two heads

                        ┌── lm_head (248,320 × 4096) ─────────────► next-token logits → text generation
Qwen3.5-9B text tower ──┤   (pristine base weights; adapter OFF)
 (bit-identical to base)│
                        └── + LoRA adapter (154 MB) ── decision head (24 × 4096) ─► calibrated typed decision
                            (adapter ON; head rows were initialised from lm_head rows)

mode	weights used	quality
Text generation	backbone + lm_head, adapter off	exactly Qwen3.5-9B: HumanEval greedy pass@1 70.7 % (116/164), completions byte-identical to the base model
Typed decisions	backbone + adapter/ + head.safetensors	the numbers in Headline results

Why the adapter is shipped unmerged: folding the LoRA into the backbone would let one weight set serve both heads, but it costs generation quality — the merged backbone with the original lm_head scores 61.6 % (101/164) on HumanEval, a 9-point drop, even though its perplexity on prose barely moves (3.15 → 3.30). Keeping the base pristine and applying the 40 M-parameter adapter only on the decision path removes that trade-off. For decision serving the adapter is merged in memory at start-up, so decision latency is the same as a merged bundle.

Usage

Option A — vLLM (recommended)

See Inference with vLLM above.

Option B — the bundled FastAPI service with a TypeSafe-Jev-style request/response schema

The training and serving code ships in this repository under code/.

hf download autotrust/JEV --local-dir JEV            # ~18 GB (backbone + lm_head) + 154 MB adapter
pip install -e "JEV/code[dev]"                        # torch ≥ 2.13, transformers ≥ 5.16, flash-linear-attention
python -m jev_judge.server --export JEV --port 18080

curl -s localhost:18080/v1/decisions -H 'content-type: application/json' -d '{
  "kind": "choice",
  "state": "SKU AX-330 stock at 8% of safety level; supplier late twice this quarter.",
  "question": "Supplier response for this scenario.",
  "options": ["issue_warning", "renegotiate", "dual_source", "maintain"]}'

{"id": "req_…", "kind": "choice",
 "options": ["issue_warning", "renegotiate", "dual_source", "maintain"],
 "distribution": [0.334, 0.142, 0.523, 0.001],
 "decision": {"noul": null, "choice": "dual_source", "score": null, "expected_score": null},
 "confidence": 0.523, "model": {"name": "jev-judge-qwen35-9b", "version": "0.8.0", "calibrated": true},
 "latency_ms": 91.3, "batch_size": 1}

POST /v1/decisions:batch ({"items": [...]}, ≤ 256, order preserved) · GET /healthz · 422 on invalid kind/options · 413 on over-long input unless "truncate": true · header X-Jev-Judge-Version · dynamic batching (≤ 4 ms window; 8 concurrent clients → 32 req/s) · refuses to start without calibration.json.

Option C — plain transformers + peft

import json, torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "autotrust/JEV"
tok = AutoTokenizer.from_pretrained(repo)
base = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda")  # == Qwen3.5-9B text model

# --- text generation: the pristine base model, no adapter --------------------------------------
msgs = [{"role": "user", "content": "In two sentences, what is safety stock?"}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = base.generate(**enc, max_new_tokens=80)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

# --- typed decisions: attach the LoRA adapter (merged here for speed) + the 24-slot head --------
model = PeftModel.from_pretrained(base, repo, subfolder="adapter").merge_and_unload()
head = load_file(hf_hub_download(repo, "head.safetensors"))
cfg  = json.load(open(hf_hub_download(repo, "judge_config.json")))
temp = json.load(open(hf_hub_download(repo, "calibration.json")))["per_kind"]
W, b = head["proj.weight"].cuda(), head["proj.bias"].cuda()

def decide(kind, state, question, options):
    letters = "ABCDEFGHIJKLMNOP"
    lines = options if kind != "choice" else [f"{letters[i]}) {o}" for i, o in enumerate(options)]
    text = f"[kind] {kind}\n[state] {state}\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"
    ids = tok(text, return_tensors="pt", add_special_tokens=False).to("cuda")
    with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
        h = model.model(**ids).last_hidden_state[0, -1].float()          # backbone only, last token
    z = (W @ h + b) / temp[kind]
    s, _ = cfg["slots"]["ranges"][kind]
    p = torch.softmax(z[s : s + len(options)], 0)
    return dict(zip(options, p.tolist()))

print(decide("noul", "Customer says the parcel arrived damaged and wants their money back.",
             "Is the customer asking for a refund?", ["false", "true"]))
# {'false': 0.009, 'true': 0.991}

options are validated: noul must be ["false","true"], score must be ["0".."5"], choice takes 2–16 free-text options. To keep both modes in one process, leave the adapter unmerged and run generation inside with model.disable_adapter():.

Why start from a pretrained LLM

A System One judge is "just a classifier", so one could train a small encoder from scratch or fine-tune a BERT-class model. autotrust/JEV starts from a pretrained 9 B LLM instead, and that choice — not the 3 GPU-hours of distillation — is where most of the result above comes from:

1. Pretraining does most of the work; distillation only sharpens. The 24-slot head is initialised from the backbone's own lm_head rows for the verbalizer tokens (false/true, 0–5, A–P), so at step 0 the head's output is exactly the pretrained model's zero-shot restricted next-token distribution (verified to |Δp| < 1e-5; measured 8.6e-07). Without seeing a single label that starting point already agrees with TypeSafe Jev 1.13 on 53 % of choice questions and reaches noul AUROC 0.82; distillation takes it to 90 % / 0.996. World knowledge, instruction reading and long-context handling are inherited, not learned.

2. Extreme sample- and compute-efficiency. Only 40.2 M parameters are trained — LoRA r=16 on the decoder projections plus the 98 k-parameter head, 0.5 % of the 8.0 B text backbone. KL 0.028 was reached after 0.49 epoch (≈ 320 k rows, ≈ 1.7 GPU-hours on one B200) and KL 0.021 after 0.93 epoch (≈ 3 GPU-hours); after the first 64 k rows validation KL was already below 0.10.

3. Transfer to unseen task families. On the Open-Jev OOD split (game-state and workflow-policy decisions never seen in training) autotrust/JEV reaches KL 0.234 and top-1 0.918 — a pretrained LLM reads the content of a new task instead of matching surface patterns of the 53 training domains.

4. Reads real, long, structured states. States in the corpus are prose, JSON game states, policy documents and retrieved passages up to 856 tokens; the backbone supports far longer. BERT-class encoders stop at 512 tokens and lack the code/JSON pretraining to parse them.

5. Reads options, not positions. With 30 % option-permutation augmentation during distillation, the top-1 flip rate under shuffled choice options is 3.9 %; the same backbone before distillation flips 38 % of the time. The pretrained attention binds the letter to the option text almost immediately.

6. Calibration falls out for free. Distilling full teacher distributions with KL (plus an ordinal RPS term for score) on a pretrained backbone gives fitted temperatures of 1.002 / 0.984 / 1.012: the model is calibrated to the teacher with no post-hoc correction (ECE 0.0007).

7. The recipe scales without code changes. The head-initialisation identity holds for any qwen3_5 checkpoint without logit soft-capping; the equivalence gate also passes on Qwen3.8-27B (4.5e-07). 9 B was kept because it already meets the distributional targets.

Evaluation details

Additional metrics (test_set_30k, temperature applied)

metric	autotrust/JEV
noul Brier score against the teacher's probability (lower is better)	0.0015
score ranked probability score (lower is better)	0.0085
Fitted temperatures noul / choice / score	1.002 / 0.984 / 1.012
Top-1 flip rate when choice options are shuffled (1,000 rows × 4 permutations)	3.9 %
Out-of-distribution split — top-1 agreement · noul AUROC	0.918 · 0.989
Throughput — batch of 128 requests on one B200	2.5 ms per decision (≈ 400 decisions/s)
Throughput — 8 concurrent clients, single requests	32 requests/s, median 250 ms

Text-generation head — no-degradation check (HumanEval, greedy pass@1, completion-style prompt)

weights	pass@1	note
Qwen3.5-9B (base)	70.7 % (116/164)	same loader and protocol as below
autotrust/JEV — generation path (backbone + lm_head, adapter off)	70.7 % (116/164)	all 164 completions byte-identical to the base model
LoRA folded into the backbone + base lm_head (not shipped)	61.6 % (101/164)	why the adapter is kept separate

Per source × primitive (test_set_30k, temperature applied)

source	kind	n	KL	top-1	ECE	noul AUROC	score MAE
yuri_v3 — synthetic operational scenarios, labelled by TypeSafe Jev 1.13	noul	8,537	0.005	0.961	0.001	0.994	—
yuri_v3	choice	8,312	0.028	0.902	0.002	—	—
yuri_v3	score	8,527	0.023	0.883	0.002	—	0.103
openjev_v2 — Open-Jev programmatic tasks	noul	1,432	0.004	0.998	0.003	1.000	—
openjev_v2	choice	887	0.176	0.857	0.020	—	—
yuri_v1 — placeholder [0.5, 0.5] labels (see Limitations)	noul	2,260	0.000	—	0.005	—	—

OOD split (13,058 Open-Jev rows from task families not in training): KL 0.234, top-1 0.918, noul AUROC 0.989; choice KL 0.351 / top-1 0.837 (game-state decisions are the hardest slice).

Choice option-permutation consistency (1,000 rows × 4 random permutations): mean max |Δp| 0.024, p90 0.055, top-1 flip rate 3.9 %.

How fast it got there (most recent first)

Validation KL on a fixed 4 k-row subset; test_set_30k metrics after calibration.

stage	rows seen	val KL	t30k KL	choice top-1	score MAE	noul AUROC	ECE
autotrust/JEV v0.8.0 — released weights (4,750 steps ≈ 0.93 epoch, LR annealed to ≈ 0.07×)	608 k	0.019	0.0210	0.898	0.103	0.996	0.0007
v0.7.0 (step 2000 + 500-step LR cool-down)	320 k	0.026	0.0276	0.884	0.119	0.994	0.0014
step 2000	256 k	0.0325	0.037	0.865	0.143	0.992	0.004
step 1500	192 k	0.038	0.040	0.861	0.151	0.991	0.0025
step 500	64 k	0.094	0.081	0.817	0.224	0.977	0.022
untrained backbone with the initialised head (reference point, not the model)	0	0.485	0.510	0.532	1.130	0.824	0.094

Annealing matters: the v0.7.0 cool-down (500 steps, lr ×0.9 → ×0.02 from step 2000) lowered KL by 25 %; continuing on the unseen remainder of the epoch with the learning rate decayed to ≈ 0.07× (v0.8.0) lowered it by another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high learning rate should be annealed.

Training details

item	value
teacher / data	TypeSafe Jev 1.13 full output distributions (as published in the corpus), SargeDev/jev-distill-corpus-v3 (740,957 rows; train 655,806)
backbone	Qwen/Qwen3.5-9B text tower only (vision tower and MTP head dropped), bf16, frozen
trainable	LoRA r=16, α=32, dropout 0.05 on in_proj_qkv, in_proj_z, out_proj, q/k/v/o_proj, gate/up/down_proj (40.1 M, shipped unmerged in adapter/) + 24-slot head (98 k, fp32, initialised from lm_head rows)
loss	KL(teacher ‖ model) over active slots + 0.5 · RPS (ranked probability score) for score
augmentation	30 % random permutation of choice options (targets permuted consistently)
batching	128 rows / step, kind-stratified (≥ 1/6 per primitive), length-bucketed, micro-batches capped at 24 k padded tokens, gradient checkpointing
optimiser	AdamW (fused), β=(0.9, 0.98), lr head 2e-4 / LoRA 1e-4, cosine, warmup 3 %, grad-clip 1.0 for 2,500 steps; then continued on the unseen remainder of the epoch (fresh AdamW state, warmup 2 %, lr ×0.9 → cosine) and stopped after 2,250 more steps at lr ≈ ×0.07 — 4,750 steps ≈ 0.93 epoch in total
label hygiene	yuri_v1 rows carry exact-uniform [0.5, 0.5] placeholder labels (137,203 rows, 100 %); down-weighted ×0.05 in training and excluded from temperature fitting
calibration	per-kind scalar temperature (L-BFGS on the calibration split, 10,954 rows): noul 1.002 · choice 0.984 · score 1.012
compute	1× NVIDIA B200 (183 GB); ≈ 1.4 h (2,500 steps) + ≈ 1.5 h (2,250 steps) ≈ 3 GPU-hours; ≈ 7–9 k tokens/s
software	torch 2.13 + cu130, transformers 5.16, peft 0.21, flash-linear-attention 0.5.2

Limitations

• autotrust/JEV mirrors TypeSafe Jev 1.13, including its mistakes. This is a distillation, not an independent judge: where the teacher was wrong or uncalibrated, so is autotrust/JEV. Published evaluations of the teacher show it is unreliable for multi-hop reasoning, arithmetic, dates, counting, and adversarial inputs; autotrust/JEV inherits all of that. It cannot generate text or explanations as a decision model; the generation head is simply the unmodified base model and knows nothing about the decisions it is packaged with.
• It inherits the teacher's blind spots, and at 9 B adds some of its own. On fresh inputs it shoves a poker spot a solver always checks (0.70, teacher 0.62), 11.5 % of its 16-option answers change with option order alone (teacher 7.0 %), and it missed a code-rule violation that the 27 B sibling catches. Prefer autotrust/JEV-27B for long option lists and code-rule checks.
• Choice agreement is capped by teacher ambiguity. The teacher's choice distributions are soft (median top-1 probability 0.70). On the 14 % of rows where the teacher's top-2 options are within 0.1 of each other, argmax agreement is near chance for any faithful mimic (0.46 where the gap is < 0.05). On teacher-decisive rows agreement is 0.954, and autotrust/JEV's argmax captures 97.7 % of the teacher probability mass a perfect mimic could (0.693 vs 0.709).
• Fixed option sets. noul and score accept only their canonical options; choice accepts 2–16 options. Inputs longer than 1,024 tokens are truncated (state only, head 60 % / tail 40 %) at serving unless you raise the limit.
• English-centric. The corpus is English; multilingual behaviour is inherited from the backbone but was not measured.
• Placeholder labels in the corpus. The yuri_v1 memory-relevance stream is 100 % exact-uniform [0.5, 0.5] — those rows teach nothing about relevance. autotrust/JEV outputs ≈ 0.5 on them by design; do not use it for memory-relevance scoring without further training.
• Not for high-stakes decisions. Use confidence gating: act automatically only above a threshold you validated on your own data, and route the rest to a stronger model or a human.

Files

model-0000{1..5}-of-00005.safetensors   Qwen3.5-9B text backbone incl. lm_head — bit-identical to the base model
                                        (bf16; GDN A_log / gated-norm weights fp32 as in the original), 17.9 GB
model.safetensors.index.json · config.json
adapter/                                LoRA decision adapter (peft format, r=16, 40.1 M params, 154 MB) — apply only for decisions
head.safetensors                        24-slot decision head (fp32): proj.weight [24, 4096], proj.bias [24]
judge_config.json                       slot layout, verbalizer token ids, template version, weights_mode=unmerged, provenance
calibration.json                        per-kind temperatures (+ fit diagnostics)
adapter_vllm/                           the same adapter for vLLM: backbone LoRA (zero-padded to r=32) + decision head as an lm_head LoRA,
                                        plus decision_head.json (head bias, verbalizer token ids)
tokenizer.json · tokenizer_config.json · chat_template.jinja
code/                                   jev_judge package, scripts (train / calibrate / evaluate / export / serve),
                                        tests, configs, DESIGN.md and all evaluation reports

License and acknowledgements

Weights: Apache-2.0 (base model Qwen/Qwen3.5-9B is Apache-2.0; training corpus SargeDev/jev-distill-corpus-v3 is Apache-2.0, its openjev_v2 stream additionally CC0). The System One framing and the noul / choice / score primitives originate with TypeSafe AI's Jev; autotrust/JEV is an independent student model trained on public data and shares no weights, code or affiliation with TypeSafe AI.

@misc{autotrust_jev_2026,
  title  = {autotrust/JEV: an open-weights student of TypeSafe Jev 1.13 for calibrated System One decisions (Qwen3.5-9B)},
  author = {autotrust},
  year   = {2026},
  url    = {https://huggingface.co/autotrust/JEV}
}