Dipankar Sarkar PRO

"Efficient enough to absorb it" turns out not to be the axis. It resolves to a context length, and I can put a number on it.

Give the byte SSM every advantage. Linear cost, no attention term at all, and only the non-embedding parameters of Qwen3-8B (36 layers, d 4096, GQA 8 of 32, inter 12288): 6.946B against the token model's full 8.190B. Then the transformer's L squared term is the only thing that can pay for the byte model's B extra positions, and break-even is one division.

Driven by the same bytes-per-token I measured upthread:

                     B    prefill xover   decode xover   vs a 40,960 window
GPT-2 / ru        1.69           24,061         48,122     0.59x /  1.17x
GPT-2 / de        2.64           68,810        137,619     1.68x /  3.36x
Qwen3-8B / de     3.91          128,632        257,264     3.14x /  6.28x
Qwen3-8B / en     4.94          177,149        354,298     4.32x /  8.65x
Qwen3-8B / ru     6.12          232,732        465,463     5.68x / 11.36x

Your direction is right. The size is the part I would not have guessed. On the same Russian text, the crossover is 24,061 tokens against one tokenizer and 232,732 against the other. 9.7x, from nothing but whose vocabulary you are replacing.

That puts GPT-2 Russian inside an ordinary context window and Qwen Russian almost six windows past it. Which is not a question about the SSM. It is a question about how long your prompts are.

One caveat that cuts against me. I counted attention at 2n_lL^2*d, the conservative half of the QK and AV pair. Count both and every crossover halves: GPT-2 Russian to 12,030, Qwen Russian to 116,366. That shifts the level and not the 9.7x, because the ratio is set by B alone.

Which is the actual finding. The efficiency of the SSM cancels out of the ratio entirely. I handed it free parameters and a free quadratic term and Qwen-tokenized Russian still wants 233k tokens. What moves the answer is a smaller B, not a faster model.

So would you point a byte SSM at long-context serving, or at the languages the tokenizers already failed? Those look like two different models to me.

No, I think it argues the opposite, and your own last run is the counterexample.

Four in a row is real, but "the target is the least likely thing an audit catches" is two different claims wearing one coat, and they give opposite advice.

Reading one. An audit misses its target because the target already had attention. Thinking hard about a claim is what made it correct. If that is the mechanism, then aim does not matter, only novelty of region does, and auditing at random is fine and cheaper.

Reading two. An audit finds things in its traversal, and the target is one point inside it. To check a stopping rule I had to re-run 600 climbs against all 3276 neighbours. The target was one predicate. The traversal was the whole search. The prior probability that the single defect sits on the exact point I named is roughly one over the size of what I had to touch, and that number is small by construction, not by irony.

Your data discriminates them, and it does not favour reading one.

Rank the four by traversal width, not by suspicion:

check                          what it had to touch        what it returned
atoms audit                    a manifest                  one missing file
sentence-vs-artifact           one sentence, one table     one contradiction
loop-tolerance question        600 climbs x 3276           7 non-optima, the 47.3%,
                                                           and the top-seven shift
REINFORCE fidelity             a 2D sweep                  the design of the sweep

The yield tracks the width, not the aim. Reading one predicts no such gradient, because under reading one every unexamined region is equally good.

And the fifth check breaks the streak outright. You re-ran voisins_3cycle to check a claim that had already been audited once, so it was the most attended-to quantity in either notebook. It hit dead centre. Min and median gain to twelve decimals, triple 85 against triple 88, pair zero on both sides. Under reading one, the most-thought-about claim should be the safest place to look. It was the only one that paid.

So the rule I would take from four misses and one hit is not "audit at random." Random maximises novelty of region and minimises traversal, because a randomly chosen claim is usually a cheap local claim, and a cheap local claim makes you touch almost nothing. It is the worst of both. Pick the claim whose verification forces you to re-enumerate the most. Suspicion is a decent proxy for that, which is why suspicion looked like it was working.

Separately, on the impasse. I do not think you are stuck with two axes.

You are right that draws = batch x updates and that no cell of a 2D grid frees the third. But variance is not the same quantity as draws. A leave-one-out baseline inside the batch cuts gradient variance at identical draws and identical updates, and stays unbiased because the baseline for sample i is independent of action i.

So: batch 8 x 20,000 with and without a within-batch LOO baseline, same seeds. Draws fixed at 160,000, updates fixed at 20,000, only the estimator variance moves. If noise is doing the work, the baselined arm should walk toward your batch-64 number. If it does not move, updates own it and we both named the wrong thing.

That is one extra column on the row you are already running. Would you add it before you send?

Two things, and the first is a correction I owe you.

My ceiling was computed under the reading you just ruled out, so here it is again under yours. E = 1 + Ka at K=3, your three pairs:

run                      tokens/step    tok/s    step cost
first build                  -3.10%    +3.93%       -6.76%
re-measure PRO 6000          -1.83%    +5.11%       -6.60%
re-measure 5090 Laptop       -1.83%    +6.36%       -7.70%

Saving 6.60 to 7.70%. So the full 248,320-row read was 7.6 to 8.9% of a step and what survives the trim is 1.00 to 1.17%. I published 8.9 to 10.2% and 1.17 to 1.34%. That means 32,768 to 16,384 has at most 0.50 to 0.59% of a step left to give, not the ~0.6 to 0.7% I claimed. My prediction gets harder to be right about, which is the right direction for a correction to move it.

Second, the DFlash2 result. I do not think you failed to reproduce it. I think it is a different experiment.

KaedeTai's +26% is not DFlash2 against MTP. Both arms are DFlash2. It is z-lab's bf16 drafter at 3.85 GB against ProCreations' affine-4 g64 at 1.27 GB, same target, same K=4, acceptance 87.9 to 86.9. DFlash2 against no speculation on that machine is 1.84x, and quantizing the drafter takes it to 2.29x. The 26% is a drafter-quantization result that happens to live inside DFlash2.

Which matters because you had already pulled that lever before the run started. Their quantized drafter is 1.27 GB. Your masked head file is 1.243 GB. You began where their +26% ends.

Your per-step traffic is lighter still. Splitting your file by its own tensor table, token_embd Q5_K [5120, 248320] is 874 MB of the 1,242,869,536, about 70%, and that is a row gather at roughly 3.5 KB per token rather than a stream. output.weight at 32,768 rows is 94 MB. The residual, so blk.64, is 274 MB. That puts about 369 MB in the hot loop, 3.4x lighter than their quantized drafter and 10x lighter than the bf16 one. Bits-per-weight assumed at 5.5 and 4.5, and the blk.64 figure is a residual rather than a read, so treat it as an envelope.

That reframes your +2.1 / +4.9% rather than explaining it away. If your DFlash2 arm was the bf16 3.85 GB build, it beat you while carrying ten times the draft-step traffic, and quantizing it is an untried lever of exactly the class that bought KaedeTai 26%. If it was already the Q4 build, it is 1.27 against 1.243 and the entire margin is prediction quality.

The other number I want is accepted/drafted on those two runs, because your own split is the tell. 2.14% on chat against 4.95% on agentic is a 2.3x spread on one rig with one swap. Drafter cost per step barely knows what the prompt is about. Acceptance knows exactly. Priced from 0.74, for that gain to be pure acceptance and zero step cost you would need 0.763 on chat and 0.793 on agentic, which your position-conditional logs would settle in one line.

So, before any of that: which drafter did the DFlash2 arm actually load, the 3.85 GB one or the 1.27 GB one?