Burton Lancaster's picture
🐝 Article: Where the Hivemind Comes From

Burton Lancaster PRO

RiverRider

AI & ML interests

Explainable AI

Recent Activity

posted an update about 1 hour ago
SWE-bench Verified scores whether an agent's patch passes the tests. It does not score whether the agent found the right file first, which is the step before it. We measured that step on all 500 instances. A 33M-parameter encoder, BAAI/bge-small-en-v1.5 at 384 dimensions, names the correct file first for 229 of 500 (0.458). Plain text search over the same checkouts gets 35 (0.070). The number that makes those readable is the floor. Hand the same index a bug report from an unrelated project and it still lands the gold file at rank 1 for 5 of 500 (0.010). So 0.458 is 45.8x chance, not 45.8x nothing. That ratio is where the argument is. recall@50 reads 0.954 and sounds like a solved problem. An unrelated report reaches the same top 50 for 0.244 of instances, so the margin over chance falls from 45.8x at k=1 to 13.6x at k=10 and 3.9x at k=50. The headline that looks best is the one carrying the least. All 500 ranked lists are published under CC BY 4.0, so the floor can be recomputed rather than believed. The article also ends with eleven corrections to claims we made earlier and got wrong, including one where the lever we proposed turned out to cost accuracy rather than buy it. We have not measured the patch step. This is the one before it. For anyone who doesn't live in SWE-bench: it gives model a real GitHub issue from a real project and scores whether the code it writes makes that project's tests pass. That single score covers two jobs, finding the file that needs changing and then changing it correctly, and only the pair is ever scored. The first step is what we measured. Nothing here writes code or runs a test, so 0.458 is a hit rate for naming the right file, not a SWE-bench resolve rate. Article: https://huggingface.co/blog/RiverRider/finding-the-file-localisation-on-swe-bench-verifie Data: https://huggingface.co/datasets/RiverRider/swebench-localisation
View all activity

Organizations

Blog-explorers's profile picture MLX Community's profile picture Hugging Face Discord Community's profile picture open/ acc's profile picture