Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
RiverRider 
posted an update 5 days ago
Post
64
SWE-bench Verified scores whether an agent's patch passes the tests. It does not score whether the agent found the right file first, which is the step before it.

We measured that step on all 500 instances. A 33M-parameter encoder, BAAI/bge-small-en-v1.5 at 384 dimensions, names the correct file first for 229 of 500 (0.458). Plain text search over the same checkouts gets 35 (0.070).

The number that makes those readable is the floor. Hand the same index a bug report from an unrelated project and it still lands the gold file at rank 1 for 5 of 500 (0.010). So 0.458 is 45.8x chance, not 45.8x nothing.

That ratio is where the argument is. recall@50 reads 0.954 and sounds like a solved problem. An unrelated report reaches the same top 50 for 0.244 of instances, so the margin over chance falls from 45.8x at k=1 to 13.6x at k=10 and 3.9x at k=50. The headline that looks best is the one carrying the least.

All 500 ranked lists are published under CC BY 4.0, so the floor can be recomputed rather than believed. The article also ends with eleven corrections to claims we made earlier and got wrong, including one where the lever we proposed turned out to cost accuracy rather than buy it.

We have not measured the patch step. This is the one before it.

For anyone who doesn't live in SWE-bench: it gives model a real GitHub issue from a real project and scores whether the code it writes makes that project's tests pass. That single score covers two jobs, finding the file that needs changing and then changing it correctly, and only the pair is ever scored. The first step is what we measured. Nothing here writes code or runs a test, so 0.458 is a hit rate for naming the right file, not a SWE-bench resolve rate.

Article: https://huggingface.co/blog/RiverRider/finding-the-file-localisation-on-swe-bench-verifie
Data: RiverRider/swebench-localisation
dipankarsarkar

Your floor is the best thing in the post, and it is not measuring chance.

Ran tools/verify_bundle.py before reading the README. 500 rows, 500 distinct ids, 500 of 500 gold ranks recomputed from the shipped candidate lists, 0 disagree, and the pooled 0.458 / 0.010 / 0.070 / 0.954 come back exactly. Walking floor_rank out to k gives your 13.6x at 10 and 3.9x at 50 too. The bundle does what it claims.

Then I asked what the floor is made of.

floor_from holds 12 distinct values across 500 rows. One unrelated report per repository, reused on every instance in it: all 231 django rows are scored against matplotlib__matplotlib-13989. So the floor is 12 query draws, not 500, and the variance between queries is unsampled.

At k=1 that shows up as fragility. Five floor events carry the whole 45.8x, and three of them are the same file, src/_pytest/unittest.py. Drop pytest, 19 instances, 3.8% of the corpus, and the margin goes 45.8x to 109.5x. Keep only pytest and it is 3.3x.

At k=50 the floor has 122 events, so it can be asked what it responds to. It responds to gold-file repetition.

                        n   floor@50   recall@1
gold file unique      180     0.128      0.517
gold file repeats 2+  320     0.309      0.425

z = -4.54 on the floor and +1.97 on recall, opposite directions. So the rows where your floor looks strongest are the rows where the encoder is doing the least, which is the reverse of an easiness confound.

Not a repo or query effect either. Inside django, one identical query text: unique 1/44 = 0.023 against repeat 38/187 = 0.203, z = -2.88. Inside matplotlib: 1/13 = 0.077 against 12/21 = 0.571, z = -2.88.

Read that as an unrelated report retrieving central files, with "is a gold file more than once" as the observable proxy for central in your own data. The floor is a file-centrality prior, not chance. Which makes recall/floor skill over centrality rather than skill over nothing. I think that is the better quantity and the one you actually measured.

It also means the ratio runs conservative where the benchmark repeats a file and optimistic where it does not.

Would a second floor query per repo move the 45.8x, or is the per-repo ranking stable enough that 12 draws is really 12 good draws?
In this post