Point 2 and point 3 are doing more work together than either does alone, and the size of that win turned out to be language dependent in a way I did not expect.

The sequence tax for going byte-level is just bytes per token. So I measured it on a parallel corpus instead of guessing: crosslingual-rule-following/model-inference-responses, 4,680 prompts rendered in en, de and ru, identical ids in all three, Llama-3.1-8B responses, one sample each.

Same text, two tokenizers:

             en     de     ru     bytes per token
Qwen3-8B    4.94   3.91   6.12
GPT-2       4.87   2.64   1.69

The byte-level tax is not a property of bytes. It is a measurement of how badly your tokenizer fits the language. GPT-2 on Cyrillic is already most of the way to a byte model at 1.69, so going byte-level there costs 1.7x the positions. Qwen on the exact same Russian text costs 6.1x.

Which is why your point 2 has to fall before point 3 can pay. In a transformer, byte-level multiplies attention work by B squared: 2.9x for GPT-2 on that Russian, 37x for Qwen. Linear cost turns that back into a factor of B.

The uncomfortable part is the direction. The tokenizer that handles Russian best is the one with the most to lose by dropping tokens. The bad tokenizer already paid the tax years ago.

So is your six-point case for one shared byte model, or does the crossover sit in a different place per language?