Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
hypothetical 
posted an update 13 days ago

The most discriminating number in this release is the KL column in release-manifest.json, and it flags one repo.

Pulled all 7 manifests, 28 variants.

M and L are the useful comparison because they are the same fixed recipe everywhere: Q4_K_M and Q8_0, no ANNA or RCO search involved. So cross-repo is apples to apples there.

Every L (Q8_0) sits at heldout_kl_mean 0.00062 to 0.00161. Except gemma-4-12B, at 0.03117. That is 19x to 50x the other six, at the same qtype and the same 8.51-8.64 bpw. Its M is 0.14189 against 0.01155-0.04488 for everyone else.

The one structural difference I can see in the manifests: gemma-4-12B is the only repo that retains zero tensors at BF16. Its L is a bare Q8_0:329, where E4B's L is BF16:85 plus Q8_0:260, and every other repo keeps 36 to 85. It is also the only one with output_embedding_mode tied_alias and lm_head_policy skip_tied.

Worth adding that the benchmark you headline cannot see any of this. On IFEval instruction-strict, 4 of the 7 recommended tiers score above their own BF16 reference (Qwen3.5-2B M at +0.84pp, E4B S at +0.48). Largest deviation in either direction across all seven is 1.68pp, and one binomial SE at n=541 is about 1.4pp. Nothing clears 2 sigma. Your own KL column separates the same checkpoints by more than 20x.

These cards are among the best I have read on the Hub, which is the only reason any of this was checkable: base revision pinned, llama.cpp revision pinned, SHA-256 and tensor inventory per file.

So: is the tied output embedding what pushes that whole path through the quantizer on 12B, and would retaining it at BF16 like the other six close the Q8_0 gap?