Your denominator answer closes the last cheap explanation, and it moves the problem off V and onto K.
Thank you for going back to the archived runs instead of answering from memory. One shared reference, one pkv, one invocation per model is exactly what was missing, because it lets me subtract your two rows against each other. That difference is the strongest object in the thread. It cancels the baseline, it cancels K, and it cancels any per-group scale or zero-point metadata, since that is additive and identical in both configs.
The K4V2 row alone is impossible, in four numbers
Qwen2.5-32B, your integers:
fp16 reference 128.0 MB
honest 4-bit K base/8 = 16.0 MB
honest 2-bit V base/16 = 8.0 MB
required 24.0 MB
K4V2 measured 20.7 MB
An honest 4-bit K on those PKVs is 16.0 MB. Your K4V2 total is 20.7, which leaves 4.7 MB for V. An honest 2-bit V is 8.0 MB. Scale metadata only pushes the requirement further up.
Read it the other way. Give V its full 8.0 and K is capped at 12.7 MB against a nominal 16.0.
The 72B says the same thing on its own numbers:
32B 72B
reference 128.0 MB 160.0 MB
K4V2 measured 20.7 26.0
V honest at 2 bits 8.0 10.0
K implied <= 12.7 <= 16.0
K nominal 16.0 20.0
K short 20.6% 20.0%
Two models, different shapes, same 20%. That is not rounding.
The difference says the opposite of the sum
Only V changed between your rows, so the delta is pure V payload:
K4V4 - K4V2 nominal V 4->2 saving excess
32B 10.2 MB 8.0 MB +27.5%
72B 12.8 MB 10.0 MB +28.0%
Your figures are three significant digits. Carrying that through, the excess band is +26.2% to +29.0%. It never reaches zero.
Now put the two facts together. Suppose every payload is off by one common factor f:
f implied by the difference 1.275 (32B) 1.280 (72B)
f allowed by the total <= 0.966 <= 0.970
Disjoint, and not narrowly. A packer that under-stores everything makes the difference smaller, not larger. So one bug cannot produce both rows. There are at least two effects and they point opposite ways.
What each side probably is
V, over by about 27%. Group metadata cancels in the delta only if the granularity matches, so the clean candidate is that your V4 and V2 paths do not share a group size. With an fp16 scale plus an int8 zero, V4 at group 32 and V2 at group 128 gives +0.5625 bits per pair, which lands inside both models' bands. No other power-of-two pair fits. I swept six group sizes on each side, 16 to 512, against three metadata layouts (fp16 scale alone, fp16 scale plus fp16 zero, fp16 scale plus int8 zero). 108 combinations, one fit. If that is what the code does, the V story is a labelling issue and not a correctness one.
K, short by 20%, and this one survives every V explanation, because it falls out of the K4V2 row with V granted its full nominal payload and nothing else assumed. You mentioned the attention importances are precomputed once. If K is 4 bits on a kept fraction and dropped elsewhere, your two models imply a keep rate of at most 0.794 and 0.800. A round 0.80 sits right at the top of both bands.
If that is the design rather than a bug, then 6.2x is real but the label is not. It would be K at 4 bits on 80% density plus V, reported as K4V2.
The one run, four integers
Not two config traces. Print K bytes and V bytes separately off the same enc object, both configs, same fixed sequence length and batch:
K4V4 -> K bytes, V bytes
K4V2 -> K bytes, V bytes
Two predictions, so this is falsifiable rather than a fishing trip:
- K bytes are identical across the two configs, since only V changed.
- that shared K number is near 12.7 MB on the 32B and 16.0 MB on the 72B, not 16.0 and 20.0.
If K comes back at 16.0 and 20.0, my arithmetic is wrong and I will say so plainly. If it comes back near 12.7 and 16.0, then enc.memory_bytes() is fine and the open question is what the K path is not writing.
Does the K quantizer use those precomputed importances to drop anything, or is every K element written?