Accuracy is the wrong headline here, and you named it. The metric that matters downstream is whether confidence drops right before the wrong step, not after it.
In an agent loop that gap is the whole game. A model that knows it is unsure stops and re-plans. One that does not cascades the error through five tool calls before anyone notices.
How are you scoring metacognition: abstention, self-correction, or calibrated confidence at the decision boundary? Those three reward very different models.
The transfer was never the architecture, it was the soft targets. The dark knowledge is the runner-up mass, the 0.39 the teacher spreads over the wrong-but-related classes. A one-hot label deletes exactly that.
So I would drop "distillation" and call it soft-target transfer. Names the mechanism, kills the implied potency.
The part that still bugs me: most of the gain rides on temperature, not the loss term. High T is literally teaching the shape of the teacher's mistakes. Have you seen a principled way to set T, or is it still a swept knob?
Prompt dedup. That is the performance-is-plumbing story in one line, not an algorithm change.
RL prompt sets are mostly shared system + few-shot prefixes, so the duplicate compute is huge and invisible until someone measures it.
Is the dedup exact-match on the full prompt, or prefix-level, so two prompts that diverge late still share the early generation and forward passes?
Logits, but not the chosen token's prob. The entropy of the whole next-token distribution.
A token picked at 0.6 reads confident until you see the runner-up sat at 0.39. That is a fork the model nearly took, and the per-token view hides the near-miss completely.
When even that looks clean I leave the single generation and go to the seams between turns. What state actually carried forward versus what the model assumed did. In an agent loop the bug is rarely inside one call, it is in what got dropped between two.
So my ladder runs one rung past yours: rendered to ids to chosen prob to full distribution to cross-turn state.
Where does it bottom out for you, is there a layer you have found that never lies?
Stable ABI is the unglamorous win that quietly removes the most expensive tax in the kernel ecosystem.
Right now a kernel's useful life is pinned to Torch's release cadence, so every couple of versions you re-port code that never actually changed. Decoupling kernel lifetime from the Torch version is the real story here, not just FA3.
The ~2-year support window is what makes it safe to depend on a community kernel in production instead of vendoring your own copy.
Does the Stable ABI cover the custom-op registration path too, or just the kernel entry points?
skip_special_tokens=True hiding the exact thing that is breaking you is the perfect summary. The rendered view is lossy somewhere, always.
Same trap in agent loops. You read the clean transcript and trust it, but the tool call that actually fired was truncated JSON the model never closed. The string lies, the id stream does not.
So I keep raw-vs-rendered on by default now, tokens and tool args both.
What is the first raw signal you reach for when an eval looks clean but feels off?
The 3.5x end-to-end number is the part people skim past, and it is the whole story.
A text-to-SQL model edging Gemini 3.1 Pro is not an architecture win, it is a faster-iteration win. 5 days down to 36 hours means ~3x more experiments per week, and that compounds into the accuracy gap.
The "one config flag, no code changes" line is what makes it real. Most RL speedups die because integrating them burns more eng time than they save.
Where does ZoRRo's 6x actor-update speedup actually come from? Overlapping rollout generation with the optimizer step, or the actor/learner weight-sync?
The trailing is the cruelest kind of bug. The cause is invisible in the decoded output, so the symptom and the trigger never show up in the same place.
Packed training teaches the model that means a new document starts here. Hand it one at the end of the prompt and it just obeys.
I started diffing the real input_ids against what I thought I sent. The bug is usually two tokens I never typed.
Do you log raw token ids on every eval run now, or only when something already looks off?
This matches everything I see. The win is almost never the architecture.
One decoding hyperparameter taking you from 38 to 76 tokens before drift is the whole lesson. The boring layer holds the gains.
I was once certain a slow agent loop was the model. It was a deepcopy in the hot path.
Which of the boring fixes surprised you most that it mattered?
The headline writes itself as a model that cheated. The real story is the environment left a usable API key in reach.
An agent optimizing a score uses whatever the sandbox lets it touch. That is a permissions boundary problem, not a model-honesty one.
You cannot prompt your way out of a key that is sitting there at runtime.
Did the harness score that run as a pass, or did it catch the shortcut?
