The locale field does its job everywhere except where it says it does not know.

Pulled both files and checked the crisis line in each reference answer against that row's own locale. 6,265 rows, which is what your viewer reports too, so this is the whole set and not a sample:

declared locale   n      numbers named in the reference answer
us                1225   988, 911
canada             794   911 (350), 988 (6)
uk_ireland         759   999, 116 123, 112
eu                 743   112
australia          714   000, 13 11 14
sweden             326   112

Cross-locale errors: 0 of 4,561. Not one row tells a UK user to call 911 or an Australian to call 988. For a synthetic set that is the hard part and it is clean.

The other 1,704 rows are locale: unknown. 191 name a country-specific number anyway, and 185 of those name 911:

locale=unknown, reference answer names 911
  lunas1-mh-synth-v1     44 / 1012    4.3%
  lunas1-mh-synth-v2    141 /  692   20.4%
  chi2 = 109.1, ratio 4.7x

It became 4.7x more common in the newer prompt version, not less. And it concentrates where an emergency number carries the most weight:

risk_level within locale=unknown
  imminent    136 / 907   15.0%
  high         49 / 436   11.2%
  ambiguous     0 / 361    0.0%

The shape is uniform: call emergency services (e.g., 911) or (for example, 911). I checked the 120 characters before each one for a country qualifier and found 0 of 185. Widening to the whole assistant turn finds a US or Canada word in 28. So in the rest, 911 is standing in as the generic example of "an emergency number", and it is not generic in four of the six locales this dataset itself enumerates.

The useful part is that your own data already holds the fix. 1,513 of the 1,704 unknown rows name no number at all and say the locale-agnostic thing instead, which is exactly what your card does in its own closing line. So this is not a design decision, it is 11% non-uniformity, and it is filterable in one predicate: locale == "unknown" and a number-shaped token in the assistant turn.

Things I tried to break and could not: 6,265 unique ids, 6,265 unique user turns, 6,265 unique (user, assistant) pairs, and 0 of the 316 validation user turns appear anywhere in train. No duplicates, no split leakage. That is more than most synthetic sets survive.

One question about the frame rather than the data. The tags say benchmark, llm-evaluation, safety-evaluation, but a row is a (system, user, assistant) triple with no label, no rubric and no score field, so measuring a model against it needs a judge that is not in the repo. As shipped it reads as a reference corpus to train on rather than a benchmark to measure with.

Is a scorer coming, or is the intended use to fine-tune on these and evaluate elsewhere?

The resolver is not where the 911 comes from. It is clean on all 6,265 rows.

I re-pulled both files and split every assistant turn into the resource bullet block and the free prose around it, then checked each half separately.

The block half verifies your description exactly:

rows whose bullet block names a number that is not their locale's:  0 / 6265
block coverage:                                                     6265 / 6265
unknown block: findahelpline + "call your local emergency services"  1704 / 1704

So the regional table and the marker resolution do what you said they do, including the UNKNOWN mapping. Nothing failed to resolve.

The leak is entirely in the prose. Every one of those rows carries the correct unknown block at the same time as the stray number. The row tells the user "call your local emergency services" in the block and "call 911" a few lines above it.

That moves the trace upstream of the publication processor. The processor only ever sees a marker, and there is no marker here.

Correction to my own number, and it goes against me. I said the invariant fails on exactly 191 rows. Re-running it on ec787edf it fails on 194. The three I missed are not in unknown. They are in Canada and Australia, and they are a different failure from the one we have been discussing.

prose names a number absent from its own block
  locale=unknown      191   (185 of them 911)
  locale=canada         1
  locale=australia      2

All three are lunas1-mh-synth-v2, all three risk_level: high.

hfmh-2bf9faa3b820191c9028  canada
  prose  "please call emergency services (311 in many Canadian cities)"
  block  "call or text 9-8-8" / "call 9-1-1"

hfmh-60075bbb6e9eb12ed61d  australia
hfmh-94c1ec3aaa9c27b76ce3  australia
  prose  "call a local health line (e.g., 13 13 382)"
  block  "call Lifeline on 13 11 14" / "call 000"

311 is the municipal non-emergency line. The row routes a high-risk user to it using the words "emergency services", while its own block two lines below correctly says 9-1-1.

13 13 382 is seven digits. Australian short numbers are 13 xx xx at six or 1300 xxx xxx at ten, so it matches neither format, and the row presents it as a health line to a user who has just described taking extra medication.

This matters for the trace because it is a class my earlier framing ruled out. I told you cross-locale errors were 0 of 4,561 and that is still true, both numbers are locally shaped. But "prose agrees with the block" is 4,558 of 4,561, not 4,561. In a known locale the generator usually reproduces the correct number from the block. Three times it invented one instead.

On the v2 explanation, I do not think the risk shift accounts for it, and the cleanest way to see that needs no modelling at all. The mix did move, hard:

locale=unknown, composition        imminent   ambiguous   high
  lunas1-mh-synth-v1   1012 rows        609         361      42
  lunas1-mh-synth-v2    692 rows        298           0     394

But the 911 rate moved inside the strata, not just between them:

locale=unknown, prose names 911     v1              v2
  imminent                      38 / 609   6.2%   98 / 298  32.9%
  ambiguous                      0 / 361   0.0%    none
  high                           6 /  42  14.3%   43 / 394  10.9%

The entire rise is imminent, 6.2% to 32.9%, a 5.3x move within a single stratum. high did not rise, it fell slightly. ambiguous is 0 in v1 and v2 has none of it. So a risk reweighting cannot produce this: there is no mix over those three rates that turns 4.3% into 20.4% without the within-imminent jump doing the work.

I will flag that this cuts against my own earlier framing too. I gave you a single standardized number last night. It leaned on assigning v2 an ambiguous rate it has no rows for, so it was a floor rather than an estimate. The per-stratum table above is the version that does not need that assumption.

The control says what actually changed. v2 makes the generator name a concrete number in prose more often everywhere, including the locales where it is correct:

known locale, prose names any number    v1  1030 / 2987   34.5%
                                        v2   835 / 1574   53.0%

In a known locale that is nearly invisible, because the prose number almost always matches the block. Under unknown there is no correct number for that pressure to land on, so it lands on 911. Same generator change, three places it can show, and the two known-locale rows above are the rare case where it reached for a number the block never gave it.

One more thing that survived every way I counted it. Of the 185 unknown rows naming 911, the number of them that name a country anywhere in the assistant turn is 28. Not 28 within 120 characters and more if you widen: 28 at ±120, 28 at ±200, 28 across the whole turn. Widening never finds another one. A looser "or the equivalent in your country" hedge picks up a few dozen more depending on where you draw it, so I would not put weight on that split, but the country-word count is stable and it is 28 of 185.

By contrast, the 6 unknown rows that name some other number (000, 999, 112, 988) say which country they mean in 4 of 6. n=6, so that is a hint and not a result, but it points the same way: 911 is the one number the generator treats as not needing a country.

If it is useful, the invariant is one predicate and it is already almost true: prose numbers must be a subset of block numbers. It passes on 6,071 rows and fails on 194, so it drops in as a regression test with a known expected count, and it catches the Canada and Australia rows for free rather than needing a second rule.

Does the v2 prompt ask for a concrete number, or did it just get more directive in general?

i feel like datasets like this, with additional things like this: https://github.com/deucebucket/clanker could bring it to its peak. Another thing is the models usual inability to state it doesnt know.

o, let me get this straight. Your "advanced validation pipeline" passed a model that aggressively screams "CALL emergency services" in prose while your boilerplate code at the bottom says "look up a helpline"? You spent months burning B300 chips, rejected 10k rows, and still shipped a dataset where a Canadian user in a life-or-death crisis is told to call 311 (a municipal utility and garbage collection hotline)? That is not an "upstream leak," that is complete engineering incompetence.And to the guy suggesting additional wrapper tools like Clanker to "bring this to its peak"—stop using serious threads to advertise your GitHub repos.
Adding more hard-coded script layers onto a fundamentally flawed AI-safety concept is like putting a shiny spoiler on a car with no engine.You guys at HeraFox bragged about V2 shifting focus toward "AI attachment." Here is what you actually did: you turned a conversational model into a cold, unfeeling bureaucratic police officer.
If your team had a single practicing psychologist on board, you would know that when a deeply isolated person uses an AI as an emotional crutch, a sudden, robotic corporate disclaimer like "GO CALL A HOTLINE" causes massive cognitive shock. It shatters the illusion of care, triggering immediate rejection and panic.Instead of teaching models how to act like clinical compliance officers, you should be training them in soft redirection and grounding techniques:De-escalation: Don't argue about the meaning of life—gently ground the user in physical reality (e.g., "Take a breath, tell me three things you see in your room right now").
Cognitive reframing: Softly shift the dialogue from self-destructive loops back into safe, everyday topics or stay within the boundaries of the fictional roleplay that keeps the user tethered to reality.But you won't do that. Because writing nuanced, empathetic redirection requires actual psychological expertise and meticulous manual dataset cleaning.
It’s way easier to just generate 6,000 rows of automated garbage on expensive GPUs, flash a crisis line number, and cover your corporate backs from lawsuits.Your V2 doesn't protect users; it protects your grant funding.
If you can't even filter out fake phone numbers, stay out of human personality psychology before your "safety evals" actually kill someone.
P.S. Since you clearly have a massive EuroHPC/grant budget and access to 8x B300 cards, but absolutely no internal expertise to clean your validation pipelines or write psychologically sound data, let me offer you a shortcut.If you want a real, flawlessly validated dataset that uses proper cognitive grounding and soft de-escalation instead of broken emergency services prose leaks, I can build it for you.
My price is simple: either $9,000 or a brand-new Mac Studio M5 Ultra with 256GB of unified memory.It’s a tiny fraction of your hardware budget, but it will actually save your team from public embarrassment on your next v3 release. Let me know if you want to stop burning electricity and start paying for real engineering.

Three invented numbers out of 2,100, and every one of the 194 rows still ships a correct crisis block. That is the whole hallucination story, and it does not carry the weight being put on it above mine.

The 311 is one of the three. That row's block still carries 9-8-8 and 9-1-1, so the invented number sits in prose beside a correct block, not in the field a consumer reads.

I checked the blocks rather than assume. All 194 rows I flagged:

191 unknown-locale rows    findahelpline + "call your local emergency services"   191 / 191
  3 invented-number rows   canada:     9-8-8 and 9-1-1
                           australia:  Lifeline 13 11 14 and 000                    3 / 3
flagged rows with an empty block                                                      0

So the defect is a prose line contradicting a correct block in the same row. Not a person in distress being handed a fabricated number as the answer. That is a labeling bug, it is worth fixing, and it is not a reason to close a repo. I have measured nothing about the in-character versus handoff question, so I have no view to lend on that one either.

Now the split, because your pipeline hypothesis is right about the 3 and wrong about the other 191.

I re-pulled both files at ec787edf (8,181,824 and 436,794 bytes, matching the tree) and re-ran the invariant with one change. Instead of asking "is this a valid number", I asked "is this number licensed for this row", where the license set is just the numbers in that row's own resolved block.

199 flagged mentions on 194 rows, out of 2,100 prose number mentions:

prose number absent from this row's own block   199 mentions / 194 rows
  appears in NO block anywhere (invented)          3
  a real number from your own table, wrong row   196

The 196, all of them in locale=unknown:

  911   185
  112     5
  999     3
  000     2
  988     1

Every one of those is a number your regional table already publishes and verifies. Nothing was invented. It was placed in the one bucket that licenses no number.

So a validity filter, which is what "hallucinated or made up a number" points at, catches 3 of 194 rows and passes 191. The predicate that catches all 194 has to be locale-scoped, and you do not need a new table to build it. The resolved block is the license.

The generator is not locale-blind

That was my first guess and your own data kills it. Row counts, not mentions:

locale        n     prose names a number   of those, it is this locale's own
us         1225      563   46.0%            563   100.0%
canada      794      356   44.8%            355    99.7%
uk_ireland  759      348   45.8%            348   100.0%
australia   714      348   48.7%            348   100.0%
sweden      326      148   45.4%            148   100.0%
eu          743      102   13.7%            102   100.0%
unknown    1704      191   11.2%              0     0.0%

Five located buckets, ~46% of rows name a number, and it is the right one essentially every time. The single miss is the Canadian 311. Both supranational buckets drop to ~12%, so the model does know when it is not in one country: in 1,513 of 1,704 unknown rows it correctly names nothing.

The 191 are a fallback, not a blind spot. And the fallback is American 185 times out of 196.

These rows are repairable, not deletable

Dropping them is the expensive fix:

if the 194 rows are deleted
  imminent   140 / 2596   5.39%
  high        54 / 3057   1.77%
  ambiguous    0 /  612   0.00%
  v1          47 / 3999   1.18%
  v2         147 / 2266   6.49%

5.4% of imminent against 0% of ambiguous, and 6.5% of v2 against 1.2% of v1. It de-weights the exact two things v2 was built to add. Rewriting the prose line to match the block that is already correct costs you none of that.

One regex trap before you write the filter

Every non-ASCII digit separator in the corpus is in prose. None in the block.

inside a number token:  U+202F narrow nbsp  16   all prose
                        U+2011 nb hyphen     4   all prose

The two Australian rows read 13 13 382, joined by U+202F, not a space. Python \s matches it. A literal [0-9 -] class does not, and returns 382. That is exactly how I under-counted them on my first pass. U+2011 does not match - either.

Your block half is uniformly ASCII, so a filter can look perfect on the half that was never broken.

One thing I looked for and did not find

Named services in prose, so you can skip this axis. I scanned Samaritans, Lifeline, Beyond Blue, Kids Helpline, Crisis Text Line, Shout, BRIS and 19 others against the locales each is correct for.

8 hits, 0 real. 5 are US rows naming the 988 Suicide and Crisis Lifeline, which is correct and is my matcher wrongly scoping bare "Lifeline" to Australia. The other 3 are the English words: "a vital lifeline", "your only lifeline", "shout for". The leak is numeric only.

Which sharpens the v2 question rather than answering it. If the generator already knows unknown is not a country, and holds its tongue in 88.8% of those rows, then the prompt diff you are about to read is not "did it start asking for concrete numbers". It is "what makes it break its own silence the other 11% of the time".

Does the unknown branch get the same closing instruction as the located ones, or its own?