Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
i64systems 
posted an update 13 days ago
Post
97
remat is no longer a one-model claim!!proved it on Qwen3-30B-A3B, K=32 of 128 experts resident, output task byte-identical to the full reference, zero bytes different *in bf16*🥰🥰 GPU comes next😈

Byte-identity was never the half that could break. Every miss demand-loads the true slice and sha256-checks it before use, so the arithmetic sees the same weights at K=32 as at K=128. The number that moves is misses per token.

I re-ran your receipts before writing this, on a clone of the repo:

  • sha256sum -c MANIFEST-SHA256.txt 279/279 OK, and a coverage diff both ways says the only unmanifested file in the tree is the manifest itself.
  • research/claims/run-checks.sh bar 1 accepted, all three malformed fixtures refused by name, ALL SHIPPED BARS GREEN.
  • verify-runlog-arith.py CHECKS 49, DISAGREEMENTS 0. verify-runlog2-arith.py on RUNLOG-2, CHECKS 70, DISAGREEMENTS 0.

So congratulations on the second model, and here is why I think the Qwen arm is the harder one.

The residency budget is the same. Your resident_bytes is linear in K (3817082880 at K=8, 7634165760 at K=16), so a 120b expert slice is 13253760 B and K=32 is 15.27 GB. Qwen3-30B-A3B at K=32 is 48 x 32 x 9437184 = 14.50 GB. Within 5 percent, on a model of nearly the same size on disk.

The routing shape is not the same. gpt-oss-120b is 36 layers x top-4 = 144 expert activations per token, which is exactly your cross-domain misses/token ceiling of 144. Qwen3-30B-A3B is 48 x top-8 = 384. At a 100 percent miss rate that is 3.62 GB per token against the 120b's 1.91 GB.

Hold your measured miss fraction constant (87.53125 / 144 = 0.60786 in corpus at K=32) and Qwen lands near 233 misses per token, 2.20 GB per token. Your 0.531246395 tok/s row implies about 616 MB/s of effective lease-plus-verify traffic, which would put the Qwen arm near 0.28 tok/s. Half.

The escape hatch is that finer granularity routes better. Your 120b numbers say it did not there: at 25 percent of experts resident the in-corpus decode mass is 0.3921440972, only 1.57x what uniform-random residency buys, and out of corpus 0.2982855903 is 1.19x. (Nice invariant, by the way: 1 - misses/144 reproduces that mass column to ten digits, so the two instruments agree.) If 8-of-128 concentrates harder than 4-of-128 did, the whole cost model tilts your way.

What did the Qwen arm's decode mis/token and topK mass columns come out at?

·

youre right that identity was never the half that could break; the number that moves is misses per token. thank you for re-running the checks!!🩵 nice to meet you!

the qwen arm, K=32 of 128, code, 32,768 tokens: 12,451,840 picks, 9,954,590 misses (79.94%), so ~304 of 380 picks per token missed and the resident mass was 20.06%- below the 25% a uniform-random 32 would give. the pool is 14,495,514,624 B (your 14.50 GB is exact). lease events 121,952, verifications 370,464, zero mismatches. 1.15 TB of verified reads in the run. leased arm 13.65 tok/s eval vs 24.59 fully resident: 1.80x the wall, 67% of it read + sha256

two caveats: the row is batch-shaped (teacher-forced over 1024-token chunks), so a layer's misses lease once per chunk and amortise: 35 MB per token effective. single-token decode with a static set leases per token, ~2.9 GB, which is the regime your 0.28 tok/s describes - your order of magnitude is right there. and the fixed set was the preregistered one, not tuned on the run's trace; on the 120b a whole-run top-32 carries ~75% of uses and a set re-chosen every 512 tokens about 80 to 84%, while a prose-chosen set is near random on code. finer routing did not concentrate harder by itself; which 32 is the lever

one more measured thing: the lease tier turned out to be a SATA SSD (546 MB/s direct read), which is your 616 MB/s. it is on NVMe now at 2.8 GB/s on the same read; the 120b work runs on that tier next. the verify is the other half of the lease and is next. i was running tests with the 120b and conflated the two at print which is admittedly pretty embarrassing. but yes, those were SATA speeds

at the end of the day we are trying to bring 120b/235b grade local models to the market, using integer pipelines and deterministic reasoning- i saw you do AI infra, and hope you too can see the value in a deterministic local agent:) the first big use will be for context continuity; particularly around agentic implementation fleets with lots of compactions and handoffs- particularly design and critical automation apparatus. keep an eye on openbob coming out!

correction, 2026-09-03: the card says "96 on NVMe". the lease tier in this run was a SATA SSD (measured 546 MB/s direct read); the card is re-cut with the right word. the identity claim is unchanged - the same bytes on any bus - and the card carried no speed number. the store is on NVMe now (2.8 GB/s on the same read). runs were completed on NVMe with our 120b work- hence the confusion. im ready to be done with 30b experiments lol

Your 3,811 decides your own closing claim, and it argues against it.

Leases are unweighted. A cold expert costs a full 9,437,184 B slice whether it takes one pick in a chunk or a thousand. So the quantity that sets your 1.15 TB is distinct slots touched per chunk, not pick-weighted top-32 mass.

On your numbers: 121,952 leases over 32 chunks of 1024 is exactly 3,811 per chunk, 79.40 per layer, out of the 96 non-resident experts in a layer. Each 1024-token window already touches 82.7% of the cold set.

Now bound what the best possible static 32 could buy. Let r be how many of your currently-resident experts get used at least once, per layer per chunk. Distinct used is then 79.40 + r, and an oracle top-32 has all 32 of its residents used, so it leases 79.40 + r - 32:

 r    distinct used   oracle leases   cut     TB
32        111.4            79.4       0.0%   1.151
30        109.4            77.4       2.5%   1.121
28        107.4            75.4       5.0%   1.092
24        103.4            71.4      10.1%   1.034
20         99.4            67.4      15.1%   0.976
10         89.4            57.4      27.7%   0.831

At r = 32 a perfect set saves nothing at all.

And r near 32 is what your own hit rate implies. 20.06% of 8,106.7 picks per layer-chunk is 1,626 picks landing on at most 32 residents, against 81.6 picks per distinct cold expert. Routing broad enough to touch 79 of 96 cold experts does not leave many of 32 residents untouched.

So I do not think which 32 is the lever. I think 1024 is. Membership only regains power once the window is short enough that distinct-used per window drops below K. Your 512-token re-selection figure on the 120b is the same pick-weighted measure, so it inherits the same gap.

Where this breaks, and it is the sharper claim not the weaker one: I am reading "a layer's misses lease once per chunk and amortise" as at most one lease per distinct cold slot per chunk. If a slice can be evicted and re-leased inside a chunk, then 79.40 overcounts distinct, U is lower, and set choice buys strictly more than this table says.

One number settles it and only you have it: distinct resident experts used, per layer per chunk. If it comes back 30 or 31, membership is worth under 3% of your bandwidth and this is a granularity problem, not a selection problem.

Your correction is worth more than you think, and not in the direction you filed it under

You filed SATA as an embarrassment. I think it is the most interesting number in the run, because your three headline figures do not fit together under a pure-disk reading.

lease volume     121,952 x 9,437,184 B = 1.1509 TB = 35.12 MB/token
leased arm       13.65 tok/s  ->  73.26 ms/token
read @546 MB/s   64.33 ms/token  =  87.8% of the entire leased wall
you attribute    67% of the wall to read + sha256, together

The read alone is 87.8%. That is already more than the 67% you give to read and sha256 combined. And it is not a serialization story either: read-then-compute would land at 9.52 tok/s, not 13.65, so the reads were overlapped and the link was running at 479.4 MB/s sustained, 87.8% duty cycle, for the whole run.

Back out the implied rate instead. Give sha256 the 23.3 ms/token that 35.12 MB costs at 1.51 GB/s, which is what I measure on my own box (1 GiB warm in page cache, coreutils sha256sum, best of four, i9-12950HX), and the read budget inside your 67% is 25.8 ms/token. That is 1.36 GB/s effective. SATA III tops out near 0.60 GB/s, so the device cannot have supplied it.

Page cache can. Your cold working set is 48 x 96 x 9.44 MB = 43.49 GB, and 121,952 leases over 4,608 cold slices is 26.47 re-reads of each one. If that set stays resident in RAM after chunk 1, the disk serves 43.49 GB and the other 96.2% of your 1.15 TB is a memcpy.

Which would mean the bus was never the bottleneck, and the tier you just corrected is not the tier the number came from.

What that does to the NVMe move

The bus goes 5.13x. The run cannot.

                        read/token   overlapped wall   tok/s
SATA  546 MB/s            64.33 ms        64.33 ms     15.55   (measured 13.65)
NVMe  2.8 GB/s            12.54 ms           ?           ?
fully resident, no lease      -           40.67 ms     24.59

24.59 tok/s is a hard ceiling set by your own resident row, so the most an infinitely fast bus can pay is 1.80x. On NVMe the read drops to 12.54 ms and stops binding entirely, which hands the run to sha256 and puts a number on the half you said is next.

And your verifications = 3 x leases + 4,608 identity is what decides it. 4,608 is exactly 48 x 96, one per cold slot, so that term is a load-time census. The 3x is the question:

1 hash per lease  (verify reuses the bytes already read)   35.12 MB/token   ->  24.59 tok/s
3 hashes per lease (each verification rehashes the slice) 106.69 MB/token   ->  14.15 tok/s

Both at 1.51 GB/s. That is a 74% gap, so the NVMe re-run measures your verify cost whether or not you instrument it. If it lands near 14 you are hashing three times and the cheapest win in the system is caching the digest, not buying a faster disk.

One smaller thing that fell out of the same arithmetic. Picks per token is exactly 380.0, not 48 x 8 = 384. The deficit is exactly 4 per token, 131,072 over the run, and your pool figure confirms all 48 layers are present. Where do the other four go?

Before the NVMe run: how much RAM does that box have, and does anything drop the page cache between chunks?

·

good questions. the receipt has the machine at 27.4 GiB guest RAM, 18.1 GiB RSS, 7–9 GiB cache, and 4–6 GiB swap; nothing dropped caches between chunks. of the 2401 s wall, reads were 1025 s at 1.12 GB/s, verify 484 s at 2.38 GB/s, and madvise 79 s. so yes: page cache materially contributed--the 1.12 GB/s exceeds the 0.55 GB/s SATA device--but the cache was only 7–9 GiB against a ~43 GiB cold set, so the split was not measured. the NVMe rerun records device bytes beside engine reads and adds an O_DIRECT control

i also agree 1024 is the lever: our 120B traces put static top-32 at ~75% of picks and reselecting every 512 tokens at ~80–84%; the planner optimizes avoided stall, not hit rate. on verification, 121,952 × 3 + 4,608 is three tensors per expert—gate/up/down--each hashed once, not one slice hashed three times. we measured verification as the bottleneck and the next row moves SHA-256 onto the CPU SHA extensions with identical digests. distinct resident experts/chunk and the 380-vs-384 pick deficit are not receipt-backed yet, so i wont guess--they need instrumentation

The split is measured. It is in the two numbers you just published, and it bounds the cache from below at half the reads.

Your own receipt, not the NVMe rerun.

read     1025 s @ 1.12 GB/s   ->  1148.0 GB moved
device   0.55 GB/s sustained  ->  563.8 GB is the MOST it could have supplied in 1025 s
cache    >= 584.2 GB          =   50.9% of every byte the engine read

That is a hard floor, not an estimate. It assumes the SATA link ran flat out for the entire read window, which it did not.

The "7 to 9 GiB against a ~43 GiB cold set" framing has the wrong denominator. 43 GiB is the set; the run moved 1148 GB through it, which is 24.9 passes over every byte. A cache holding 16 to 21% of a set that is re-read 25 times under a skewed access pattern does not serve 16 to 21% of the reads.

And you handed me the skew in the same message: static top-32 of 96 takes ~75% of picks. Two independent receipts, the bandwidth excess and the routing distribution, land in the same place.

Your 3x answer settles it, and it kills my lever, so let me say so plainly.

verify  484 s @ 2.38 GB/s  =  1151.9 GB
read   1025 s @ 1.12 GB/s  =  1148.0 GB
ratio                      =  1.0034

Every leased byte is hashed exactly once. Three tensors per expert, gate/up/down, three hashes because three objects. The 74% gap I put in front of you resolved to the 1-hash branch, and "cache the digest" was the wrong recommendation. There is nothing redundant to remove.

But the next row will not pay what it looks like it pays.

I measured the SHA extensions delta on my own box with a negative control, because I did not want to guess at it. i9-12950HX, sha_ni present, OpenSSL EVP, 16 KB blocks:

SHA-NI enabled                          1.772 GB/s
SHA-NI masked off (leaf7 EBX bit 29)    0.502 GB/s
speedup                                 3.53x

Your verify path already runs at 2.38 GB/s. That is 1.34x above this box's accelerated single-core rate and 4.74x above its software rate. You are not sitting on the un-accelerated side of that 3.53x. Either your build already dispatches to SHA-NI, or the hashing is threaded, and in both cases the headroom is a fraction of what the move looks like it offers.

Price the ceiling anyway. Hand verify the full 3.53x for free:

484 s  ->  137 s
saves  347 s  =  14.4% of a 2401 s wall

And verify is not the bottleneck on this receipt.

read    1025 s
verify   484 s     read is 2.12x verify

Run the counterfactual you are about to run. If the reads had come off the device cold at 0.55 GB/s, they take 2087 s instead of 1025, the wall goes to ~3463 s, and verify's share falls from 20% to 14.0%. The cache is what made verification look like the bottleneck. Remove the cache and it is less of one, not more.

One thing I cannot resolve from the receipt, and it decides the rest.

wall          2401 s
read          1025 s
verify         484 s
madvise         79 s
                ----
accounted     1588 s   66.1%
unaccounted    813 s   33.9%   =  24.81 ms/token

813 seconds is the second largest line item in the run and it has no name.

Are those three meters additive slices of the wall, or concurrent instruments? If additive, a third of the run is somewhere neither of us has looked, and it is bigger than verification. If concurrent, then 1588 s of meter time inside a 2401 s wall means the overlap is poor and the 20% you assign to verify is not a wall-time share at all.

Which is it?