Burton Lancaster PRO
AI & ML interests
Recent Activity
Organizations
https://blackwindow.xyz
Open the site, pick a model (about 0.6B to 8B), hit Load. The weights run in that tab, on that computer. After they load, the network can drop. The context window is a working set, auto-sized to that device, up to ~32K tokens.
Behind the window is the Weave. Every file, picture, recording, link, lookup, and reply is embedded as it arrives. Drop in audio and it is transcribed. Drop in an image and it is described. A question pulls the nearest passages back as notes. A long document is walked once so later questions can use the whole file, not the first pages.
Nothing leaves that tab unless you turn on live lookup or connect a rented GPU box, and the chat says so each time. Prompts can go to the box. Files and the Weave stay in the tab.
Console on that page: bw.ask, bw.search, bw.digest, bw.notes. A local relay exposes /v1/chat/completions on localhost so other tools on the same computer can talk to the tab. The tab polls the relay. That is the boundary.
Not a server with a policy. Your hardware, a window, a Load button.
If on mobile add to home-screen for best performance. If you break it lmk. It can serve a few hundred of you at a time before I have to buy a real server.
Building it turned up three things.
We graded 330 models on Korean and two axes collapsed.
Honorifics โ only 8.5% earn an A
Knowledge of Korean institutions โ 9.4%
Every other axis sits above 31%
Fluency hides it. A model can write clean, natural Korean and still attach an honorific to a coffee cup. Fluent and wrong at the same time is worse than obviously broken, because nobody catches it in review.
A 2023 model beats the 2026 flagships. gpt-3.5-turbo-16k scores a perfect 3.00. Korean cannot be inferred from release date, parameter count or English benchmarks โ it has to be measured, per model.
Quality, value and speed are three different models. Across five axes, the same model almost never takes two columns.
425 models, latency measured on 329 on a paid API, Korean graded on 330. Three languages, three currencies, daily refresh, open API, no key.
๐ https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard ๐ ginigen-ai/open-router-leaderboard
We measured that step on all 500 instances. A 33M-parameter encoder, BAAI/bge-small-en-v1.5 at 384 dimensions, names the correct file first for 229 of 500 (0.458). Plain text search over the same checkouts gets 35 (0.070).
The number that makes those readable is the floor. Hand the same index a bug report from an unrelated project and it still lands the gold file at rank 1 for 5 of 500 (0.010). So 0.458 is 45.8x chance, not 45.8x nothing.
That ratio is where the argument is. recall@50 reads 0.954 and sounds like a solved problem. An unrelated report reaches the same top 50 for 0.244 of instances, so the margin over chance falls from 45.8x at k=1 to 13.6x at k=10 and 3.9x at k=50. The headline that looks best is the one carrying the least.
All 500 ranked lists are published under CC BY 4.0, so the floor can be recomputed rather than believed. The article also ends with eleven corrections to claims we made earlier and got wrong, including one where the lever we proposed turned out to cost accuracy rather than buy it.
We have not measured the patch step. This is the one before it.
For anyone who doesn't live in SWE-bench: it gives model a real GitHub issue from a real project and scores whether the code it writes makes that project's tests pass. That single score covers two jobs, finding the file that needs changing and then changing it correctly, and only the pair is ever scored. The first step is what we measured. Nothing here writes code or runs a test, so 0.458 is a hit rate for naming the right file, not a SWE-bench resolve rate.
Article: https://huggingface.co/blog/RiverRider/finding-the-file-localisation-on-swe-bench-verifie
Data: RiverRider/swebench-localisation
The Weave never evicts. Only the window does, and it is rebuilt from scratch every turn rather than trimmed. Every turn is embedded as it ends, so it stays retrievable as a memory:turn <n>#<i> passage for the life of the session. What changes is whether a turn sits in front of the model verbatim or comes back as a retrieved passage.
Sizing happens per turn, in pickWindow. The cap is min(HISTORY_BUDGET, promptBudget() - NOTES_RESERVE() - (this message + 700) - attached file lengths), where promptBudget() is (CTX - REPLY_TOKENS - 48) * 3.8 on desktop and * 3.3 on mobile, and NOTES_RESERVE() is 2800 desktop, 1400 mobile, which is what six notes take on a desktop and four on a phone. The last turn is always kept, because a follow-up points at it, and then it walks backwards adding older turns while they fit under the cap.
The eviction order that matters is in fitPrompt, and it is not recency. Files that fit whole go first, since they are the question's material. Then the retrieved notes. Then a long file's opening last, with whatever is left. The opening sits below the notes because of a failure we measured: asked for a code word on line 20000, the model answered from line 0, because the opening had crowded out the passage that actually contained it. An opening is context, not the answer.
Two details that were not obvious until they bit. Room left over after the notes goes back to the transcript, most recent first, and any recalled memory:turn note whose turn has just returned to the window is dropped from the note set, because otherwise the same turn is in the prompt twice and the retrieved copy is the stale one. And turns stay in RAM for three budgets' worth after they leave the window (trimHistory, while size > HISTORY_BUDGET * 3), so a later turn with few notes can take them back cheaply; past that they live on as memory only.
On mobile Safari we landed where you did. WebGPU decode runs in llm_worker.js for exactly the reason you hit: a long token loop on the main thread trips the iOS watchdog that surfaces as "a problem repeatedly occurred". Media decode is a second worker, and it loads a plain build under WebKit.
The check grows. All three of your points were the same failure the check was
written to catch, one level up, and it is rewritten rather than patched.
Coverage is now the default. It walks to any object holding a numeric oracle
at any depth instead of matching d["arms"], and takes the bounded set as every
numeric sibling not exempt for a stated reason instead of an allowlist. Your
count was 266 of 1,373. It now judges 2,501 cells across 23 files.
Per file, old against new on the shapes you listed:
verifier/exec_guided.json 0 -> 148
verifier/results.json 0 -> 666
hivemind_census.json 0 -> 60
he_k32_k_scaling.json 0 -> 72
exec_guided is the one that makes your second point concrete. It was on the
allowlist as a file name, its columns are exec and exec_verifier, and
neither was on the list, so the file printed a line and judged nothing. All
five dead names are gone; there is no list of judged columns any more, only a
list of exempt ones, and each carries its reason: subset rate, count, model
size, difference between two columns, ranking score, or outside [0, 1].
The closing line now names its own scope:
scanned 746, carried an oracle 22, judged 2501 cells, skipped 930
no column scored over all problems exceeds its oracle in any file scanned above
--verbose prints the 930 by reason. Largest are 936 floor, which is now
judged rather than skipped and is bounded by construction, 188consensus_on_covered, 82 n, 81 gain_over_floor, 78 on_resolved_only, 42headroom_captured.
Your subset-rate counts reproduce exactly on the current files: verifier
chat_consensus 6, chat_consensus_mbpp 1, consensus 12, consensus_mbpp 1;
verifier_1024 chat_consensus 3, consensus 7, consensus_mbpp 1. Zero violations
on any all-rows column, which is also what you found by hand across all 660
cells the old shape could not reach.
Your third point paid for itself immediately. artifacts/nla/q4/ one_forward_pass_probe.json has been unparseable since 24 August: Python
implicit string concatenation inside a .json file, from a paste. The old
check caught the exception, returned None, printed nothing, and closed with
"no all-rows selector exceeds its oracle in any current file", exit 0. Three
weeks of runs called the repo clean without reading it. It now prints
UNREADABLE and the file is fixed; all 1,035 artifacts under artifacts/ parse.
paper_hivemind 881 said "one pass over every results file" when the pass was
two shapes wide. Rather than narrow the sentence, the paragraph now states what
the first version did, 266 cells and a silent return, and what the current one
does with the scanned/judged/skipped counts above.
Finding the File: Localisation on SWE-bench Verified, Measured Against a Permuted-Query Floor
https://blackwindow.xyz
Open the site, pick a model (about 0.6B to 8B), hit Load. The weights run in that tab, on that computer. After they load, the network can drop. The context window is a working set, auto-sized to that device, up to ~32K tokens.
Behind the window is the Weave. Every file, picture, recording, link, lookup, and reply is embedded as it arrives. Drop in audio and it is transcribed. Drop in an image and it is described. A question pulls the nearest passages back as notes. A long document is walked once so later questions can use the whole file, not the first pages.
Nothing leaves that tab unless you turn on live lookup or connect a rented GPU box, and the chat says so each time. Prompts can go to the box. Files and the Weave stay in the tab.
Console on that page: bw.ask, bw.search, bw.digest, bw.notes. A local relay exposes /v1/chat/completions on localhost so other tools on the same computer can talk to the tab. The tab polls the relay. That is the boundary.
Not a server with a policy. Your hardware, a window, a Load button.
If on mobile add to home-screen for best performance. If you break it lmk. It can serve a few hundred of you at a time before I have to buy a real server.
Both, and both are in now. Commit 0480553c on space-bacon/SRT, card updated on RiverRider/srt-hivemind.
You were right that 879-880 read as shipped when it was a recommendation. As of d56e1a8f the fix was in consensus_select.py and the check was not anywhere.
Per arm: consensus_select.py and chat_consensus.py now raise on consensus, consensus_strict and chat_consensus above the arm's own oracle, so a bad run cannot write a file. Repo-wide: scripts/check_oracle_bound.py is one pass over every results file under artifacts/nla with per-arm oracle columns. It is column-scoped as you specified: the columns scored over all problems are held to the bound; consensus_on_covered and on_resolved_only are declared subset rates and are reported beside the oracle, not judged. Files with a SUPERSEDED key are printed as such and cannot fail the run.
Its output on today's files is your table:
ok verifier/consensus.json 72 bounded cells 12 subset-rate cells above oracle
ok verifier_1024/consensus.json 72 bounded cells 7 subset-rate cells above oracle
ok verifier_1024/consensus_mbpp.json 20 bounded cells 1 subset-rate cell above oracle
ok verifier_1024/chat_consensus.json 36 bounded cells 3 subset-rate cells above oracle
no all-rows selector exceeds its oracle in any current file
The 12, 7, 1 and 3 are the counts you found, on the columns you named. Zero violations on any all-rows column, which is what the corrected files should show and what the unguarded invariant would have muted.
The README mismatch is fixed in the card and in the paper: 12 of 36 is the 192-token file, 7 of 36 the 1024-token file, 1 of 10 MBPP. The 192-token pools stay in the repo as superseded evidence; nothing in the paper reads from them any more, and the check treats them accordingly.
Your inverse-Simpson result stands as the cleanest statement in this thread of what the template is not doing, and the corrected selector numbers point the same way. Thank you for pulling the commit and grepping rather than taking the sentence at its word.
It falls back to a pick. srt_select.select() returns replies[0] when nothing runs, so the deployable number is the fallback bound. I now report the strict bound beside it.
You are right, and it reproduces to the digit. consensus_select.py averaged the covered picks and subtracted an all-rows floor from them. 12 of 36 arms beat their own oracle, coverage rank-correlates +0.91 with floor (Pearson +0.79), and your two bounds on the 192-token pools come back 0.3760 and 0.3902. Everything else in the repo (exec_guided_select, verifier_select, chat_consensus, pooled_select, the demo bank) was already scoring every problem. This was one read.
Corrected, every problem scored, unresolved pools falling back to reply 0 as the shipped selector does:
| pools | covered-only | all problems | strict | share of gap |
|---|---|---|---|---|
| HumanEval, 192 tok | 0.4426 | 0.3762 | 0.3760 | 82.9% -> 61.4% |
| HumanEval, 1024 tok | 0.6301 | 0.5854 | 0.5840 | 60.8% -> 44.1% |
| MBPP | 0.8174 | 0.8094 | 0.7962 | 58.1% -> 53.4% |
The 1024-token pools are the ones Section 5.4 now carries. The 192-token matrix you read was regenerated the day before your comment after a truncation problem surfaced (43% to 80% of instruct-arm candidates were cut off mid-function), which is why your numbers and the current table differ before this correction. On the 1024 pools the bias is smaller because coverage is higher, mean 148.5 of 164 against 133.6, and no arm exceeds its oracle after the fix.
Agreement stays third of five reads on both benchmarks, behind example-filtering by 6 points on HumanEval and 4 on MBPP. Your formatted-versus-unformatted point holds: on the 1024 pools the captured split goes from 0.887 / 0.613 to 0.489 / 0.405. The verifier's deficit to agreement shrinks from 0.1199 to 0.0752, and the agreement decay slope on MBPP moves from -0.0815 to -0.0830 per decade.
Paper 5.4, the dataset card, srt_select, fig 3 and the census are corrected and pushed to RiverRider/srt-hivemind. The old files are kept as *_covered_only_superseded.json with a SUPERSEDED key, same convention as union_ceiling.json. You are credited by name in the limitations, where this is now the fourth accident that flattered us, and in the acknowledgments. The check that would have caught it costs one line, no arm may exceed its own oracle, and it is in the script now. The article body is being replaced with the corrected paper.
Your inverse-Simpson result is the cleanest statement in this thread of what the template is not doing. 7.9 of 8 effective distinct candidates in both families and zero byte-identical pools means the chat format is not deduplicating anything. It is making the model right more often, which is where the corrected numbers point as well.