You are right on every point, and the number you asked for is worse than the one
it replaces. Both halves took a re-encode, so this is late rather than quick.
Your chain, checked link by link
Every step reproduces. cross_runtime_browser_rung.json does declare gallery
1,000 and captions 5,001. Its 0.2300 is the f32 row of
gallery_precision_cost.json. That file's int8 row is eval_only_1000 in
gallery_scale_cost.json to the digit, and that file's full_123287 is
0.0148 / median 596, which is old_head_for_reference, n_train: 4000.
The two LFS identities hold: the Space's anchor.bin and
anchor_candle_q4_text.bin are ecc143e0, and the Space's head.safetensors
and head_v3.safetensors are 09c7322a. The two reports are one blob,
06e56278, 564 bytes.
One thing you could not see from outside, and it strengthens your reading: the
head file that run actually loaded, artifacts/local/browser/head.safetensors,
hashes 0663ff9b and matches neither the shipped head nor the 118k head. It
is a third set of weights, exactly as your chain implies.
The val-pool label is wrong too, and your arithmetic beat our own metadata to
it. browser_head_118k.pt carries n_eval_images: 5000 and split: train on COCO train2017, evaluate on all of val2017.
Does head_v3 have a number of its own
Yes, and it is 0.1108. head_v3.safetensors is the fp16 export of
browser_head_v2_best40.pt: txt.weight, txt.bias and mu_txt all match
under fp16 round-trip, while the 118k head misses on all three by up to 0.56.
And that checkpoint's own meta carries val_only: {i2t 0.5348, t2i 0.3985},
the rows in the report.
So the shared blob is not a stale copy. It is one measurement of one set of
weights under two filenames, and arm: best40_final is accurate rather than
left over. "v3" is a deployment name for the best-40 checkpoint. That naming
cost you an afternoon and it is being fixed in the Files table.
The published table is correct and mislabelled
Re-run in its own configuration, 4k head, last-token pooling, pool 1,000:
| arm |
published |
re-run |
| pytorch fp16 |
0.2300 |
0.2300 |
| candle Q4 as-is |
0.0154 |
0.0154 |
| candle Q4 + anchor |
0.1952 |
0.1952 |
Bit-identical, not just to four places: 0.015396920615876825 and
0.1951609678064387 in both files. That table was measured correctly and
computed correctly. What was wrong is that it was presented as the
deployment's, while describing an earlier head, a different pooling, and a pool
123x smaller.
What the shipped head actually does
Shipped head, shipped 123,287-image gallery, mean-pooled as the browser pools,
same 5,001 captions:
| runtime |
t2i R@1 |
R@5 |
R@10 |
| pytorch fp16 (reference) |
0.1092 |
0.2442 |
0.3307 |
| candle Q4_0, head as-is |
0.0000 |
0.0000 |
0.0002 |
| candle Q4_0, + 4 KB anchor |
0.0350 |
0.1062 |
0.1518 |
Recovery is 32%, not 85%. Without the anchor the read-out is at zero, so it
is still the difference between a working system and a dead one, but it is not
close to sufficient, and the card no longer says it is.
Four things make that number checkable rather than asserted. The fp16 arm
independently reproduces the headline (0.1092 / median 36 against 0.1108 /
median 33). The head is byte-identical to the Space's. The index is the shipped
SRTIDX02. And the anchor the run computed is byte-identical to the shipped
anchor.bin, cosine 1.0, max abs diff 0.0, so the tab is running the anchor
that was measured.
The gap between 85% and 32% is dominated by pool size rather than by anything
being broken, and our own published data shows it with no quantization
involved: the v2 head scores 0.4959 against 1,000 images and 0.0628 against
123,287, fp16 on both sides. The residual the anchor cannot remove is cheap
against 999 distractors and expensive against 123,286.
Two harness defects, because they are the transferable part
The first measurement I ran returned 0.0136 recalibrated and I nearly posted it.
browser_rung's Tap::new defaults to Pooling::Last, and the shipped head is
mean-pooled, so it was scoring the wrong input entirely. A mean vector cannot
repair a pooling error, which is what gave it away. The Pooling enum's own doc
comment warns about this three lines above the constructor. Pooling is now an
explicit flag and is printed and recorded.
The second is smaller: the prebuilt harness on this machine only understood
SRTIDX01 while the shipped gallery is SRTIDX02. I initially read that as
evidence the harness had never been pointed at the deployed index. That was an
inference from a stale local build rather than a measurement, and I had already
overwritten the timestamp that would have tested it, so I am dropping the claim.
The artifact says "gallery": 1000 on its own and needs no help from me.
Card
Anchor table replaced with the deployment measurement and the 85% relabelled as
the 4k head at pool 1,000. The int8 rows are scoped to the question they answer,
which is that quantizing the gallery costs nothing on a fixed head and pool. Val
pool corrected to 5,000. The pool-size sentence is attributed to the v2 head.
Artifacts: cross_runtime_browser_rung_123k.json (deployment, with pooling,
head sha256 and index recorded), browser_fp16_reference_123k.json,
cross_runtime_browser_rung_repro.json (the bit-identical reproduction), and
scripts/browser_anchor_reference.py. --mean and --dump-anchor are in
browser_rung so the anchor identity check is reproducible rather than
something you have to take from me.
What I do not know yet
The rank distribution of the anchored Q4 arm. R@1 and R@10 are above but not
median rank, which is the number the card leads with for fp16 (33 of 123,287)
and the one that says whether the deployed demo is merely worse or actually
unusable for its own purpose. That is running next.
And the standing question underneath it: this drift is structured rather than a
translation, which is presumably why 4 KB of mean only buys back a third of it
at scale. Fitting the text head on states encoded by the deployment runtime,
instead of fitting on fp16 and patching, is the obvious test of whether the 68%
is recoverable at all. That is a training run on the 0.6B, so it is cheap.
Thank you for reading the files rather than the post. Every number you
questioned turned out to be real, and every label on them turned out to be
wrong.