Gjxv PRO

Thanks, I hadn't run that comparison before, so I did it now on the per-case logs behind that update.

Raw errors do repeat. 128/335 and 152/335 episodes have an error per seed, 92 shared (about 58 expected if independent, φ 0.42), and 238 calls wrong in both seeds vs about 76 expected.

Confident mistakes don't. Past each seed's abstention threshold, s42 made 2 accepted-and-wrong calls and s43 made 39 over 17 episodes, with one episode in common. Weak evidence at n=2, but it points at s43's own calibration, not at screens both models fall for.

The shared errors do have a direction. 172 of the 238 are mid-episode screens accepted as "task complete", and they sit on the templated tasks (the social-media one is 24% of the set but 38% of shared errors). The two I inspected were busy content screens, a video feed and a lecture slide, with no trace of the goal's artifact yet. Fits the reading that the goal text is under-used.

Per-case JSONLs available if you want to dig in.

Checked it, and you're right about how the script works. Here's the loop you described, on the frozen logits (T = 7.653, tau = 0.79, Clopper-Pearson 95% upper bound in brackets):

• split (n=480): coverage 0.900, risk 21/432 = 4.86% [7.3%]
• known (n=48): 0.979, 1/47 = 2.1% [11.3%]
• new (n=54): 0.944, 1/51 = 2.0% [10.5%]
• mono (n=60): 1.000, 2/60 = 3.3% [11.5%]
• e, fresh OOD extremes (n=202): 0.960, 16/194 = 8.2% [13.1%]
• task (n=300): 0.683, 15/205 = 7.3% [11.8%]

So no, the 5% budget doesn't hold on the OOD extremes. The gate keeps 96% of those rows and 8.2% of the kept ones are wrong. The ECE gap you mentioned was already showing this. Small correction: 0.875 is the E-extrap pair score; row accuracy on that set is 0.906.

The 4.86% is an in-sample number. Freezing the rule in advance doesn't fix that, since the value still comes from the 480 rows that chose tau. I've added the per-set check as scripts/e36_heldout_policy.py (it reads the shipped policy and changes nothing). The model card now has the per-set table with intervals and says the 5% budget is only validated in-sample on in-range numeric questions:

https://huggingface.co/Gjusev/MK-SH-1
https://github.com/Gjusev/MK-SH-1/commit/9e1a226e

Thanks for reading the code 🫥 . This is the kind of check I should have run before quoting the number.