Mati-14B

Mati-14B is a decision scorer. You give it one instruction or state and a declared set of options, and it returns a probability distribution over those options. Like Kev, it is a soft-tuned LoRA adapter plus a pointer head, here on Qwen/Qwen3-14B-Base, tuned specifically for ranking use cases.

I have ported Mati to the same contracts as Kev (the pointer head and the serving contract), so it loads with Kev's library as-is through kev.serve. No Mati-specific code is needed for serving, and any TypeSafe-compatible client can call it through kev.serve.

Use

git clone https://github.com/jaredpalmer/kev && cd kev && uv sync --extra serve
uv run --extra serve python -m kev.serve --run PSanni/mati-14b --port 8011

from typesafe_sdk import TypeSafeClient
client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8011", model="kev-latest")

Note: kev.serve exposes each checkpoint under the model name kev-latest.

Results

I have benchmarked it on Kev's out-of-domain read for an apples-to-apples comparison. The training and calibration data, however, are different. Every column scores the same items. Each model is compared as it is served. Mati-14B and Kev-9B apply the temperature stored in their checkpoints. Jev is scored on the probabilities its API returns. Results on a custom out-of-domain set will be added here soon (waiting for TREC numbers).

	Mati-14B	Kev-9B	Jev
accuracy	0.803	0.822	0.857
Brier score	0.276	0.264	0.211
expected calibration error (ECE)	0.066	0.042	0.049
negative log-likelihood	0.476	0.484	0.701
mean confidence	0.867	0.832	0.903
confident errors (p ≥ 0.9 and wrong, share of all questions)	4.1%	4.0%	3.7%
accuracy when p ≥ 0.9	0.935	0.932	0.948
coverage at ≤ 5% error	0.59	0.45	0.70
coverage at ≤ 1% error	0.37	0.11	–
option-order flip rate (36 questions)	2.8%	2.8%	0.0%
minimal pairs, both siblings correct (64 pairs)	0.781	0.828	0.859

Definitions:

• ECE is the average gap between stated confidence and observed accuracy across confidence bins.
• Coverage at ≤ x% error is the largest share of questions you could accept automatically, taking the most confident first, while keeping the error rate among accepted answers at or below x%. It is an in-sample maximum on these items, not a guarantee on new data.
• Option-order flip rate is the share of Choice questions whose answer changes when the options are shuffled.
• Minimal pairs are two versions of one case in which a single sentence changes the correct answer; the pair counts only when both are answered correctly.

Accuracy by source (Mati-14B / Kev-9B / Jev; 80 questions each unless noted):

source	Mati-14B	Kev-9B	Jev
SciQ	0.988	0.963	0.988
QNLI	0.900	0.925	0.925
PAWS	0.762	0.762	0.787
MMLU	0.750	0.738	0.900
TweetEval-offensive	0.725	0.775	0.812
Emotion	0.562	0.600	0.588
deadline, date arithmetic (40)	0.825	0.800	0.925
authorization (40)	1.000	1.000	1.000
held-out policy: (A or B) and C (32)	0.906	0.906	0.906
held-out policy: (A and B) or not C (32)	0.719	0.875	0.969
held-out policy: if A then not B else C (32)	0.844	0.906	0.781

Calibration is built in. head.pt carries a temperature (T = 1.87), as in Kev's contract. The pointer head divides its logits by this temperature at inference, so every loader (kev.serve, kev.benchmark, your own harness) gets calibrated probabilities by default.

Evaluation

Reports come from kev.benchmark with logits recorded and the checkpoint's temperature applied. All 764 of 764 transfer records were evaluated, with none rejected or truncated. Kev-9B's numbers are from its published trial result at its shipped temperature on the same partition (same sha256). Jev's are from the Kev repo's scored run of Jev on the same partition. Scored question counts exclude the robustness variants (options permuted, none-of-the-above inserted), which are reported separately.

License

Apache-2.0 for the adapter and head. The Qwen3 base is Apache-2.0.