Fixing Silent Token-ID and Logprob Loss in llama.cpp Server, Measured with MURPHY
TL;DR
llama.cpp's server silently dropped or mislabeled token IDs and logprobs whenever one character was split across several tokens. The generated text was always correct, so nothing looked wrong.
- What broke: the server holds back text until a UTF-8 character is complete. Token IDs and probability records were attached per released text chunk instead of per sampled token, so tokens inside a chunk lost their IDs, their logprobs, or both.
- How much: on emoji-heavy output with Qwen3.5-9B, every unpatched response was affected, and 22,500 of 90,380 probability records went missing per endpoint and mode.
- Where: native
/completion,/v1/completions, and/v1/chat/completions, in both streaming and non-streaming modes. - Since when: probability records since October 2023, about three years; streamed token IDs since token output was added in December 2024.
- Fixed by: PR #31244, in two commits: one for token IDs, one for probability records. On the patched build, MURPHY found zero record defects across 24,000 requests, with unchanged instruction-following scores.
If you collected token IDs or logprobs from llama-server, especially on non-English text, emoji, or chat streaming, check your data. The next section shows how.
Should you check your data?
Your data may be affected if all of these hold:
- It came from llama-server, on a build before the fixes.
- You kept token IDs (
tokenson native/completion, populated wheneverreturn_tokensorstreamis true) or logprobs (n_probs,logprobs,top_logprobs). - The output contained characters split across several tokens: non-English scripts, emoji, symbols, smart quotes, arrows, or box-drawing characters from terminal output. Chat streaming was affected even on English text.
What it affects depends on how you used the data:
| Use | Affected? |
|---|---|
| SFT on text, retokenized from the output | ✅ No: the text was correct |
| Pass/fail evals scored on output text | ✅ No |
| RL reusing stored logprobs (importance ratios, KL) | ❌ Yes: misaligned logprobs corrupt the training signal |
| Distillation from top-k candidates | ❌ Yes: candidate records and byte labels were affected |
| Calibration, uncertainty, or perplexity from logprobs | ❌ Yes: biased toward non-English and emoji-heavy text |
| Pipelines that rebuild text from token IDs | ❌ Yes: dropped IDs corrupt the text itself |
Three cheap checks, no rerun needed:
- Count check: the number of logprob records should equal the number of token IDs, and on chat,
usage.completion_tokens. - Byte check: decode each record's token ID with the model's tokenizer and compare with that record's
bytes. Don't just concatenate the bytes and compare with the text: the bug moves a whole character's bytes onto one record, so the concatenation still matches. - Reference check: for a sample, rerun the prompt non-streaming at temperature 0 on a patched build and compare token by token.
How the bug works
The server reported each token only when it released that token's text, so tokens inside a split character lost their reports.
Root cause
The server sends only whole UTF-8 characters. When a character spans several tokens, it holds the text back until the last token arrives. Token reporting was tied to that text release, so it followed the text, not the tokens.
Two defects
| Token IDs | Probability records | |
|---|---|---|
| What went wrong | Held-back tokens' IDs were dropped | Saved only after text processing: held-back tokens got no record, and the surviving record carried the whole character's bytes |
| Where | Native streaming only; non-streaming kept every ID | Every mode, on both endpoints |
Worked example: A🙂B
Take A🙂B on the bisect's test model, whose vocabulary encodes each of the emoji's four UTF-8 bytes as its own token: six tokens in all. Before the fix, non-streaming returned all six IDs but only three records, and streaming returned three of each.
| Sampled token | Bytes | Text released? | Baseline token ID (streaming) | Baseline logprob record | Patched |
|---|---|---|---|---|---|
A |
41 |
Yes | Reported | Reported | Both reported |
🙂 byte 1 |
F0 |
No, held | Dropped | Dropped | Both queued |
🙂 byte 2 |
9F |
No, held | Dropped | Dropped | Both queued |
🙂 byte 3 |
99 |
No, held | Dropped | Dropped | Both queued |
🙂 byte 4 |
82 |
Yes, as 🙂 |
Reported | Reported, labeled F0 9F 99 82 |
All four sent with 🙂 |
B |
42 |
Yes | Reported | Reported | Both reported |
Qwen3.5 splits characters differently, but the mechanism is the same: any character that spans several tokens loses all but its last token's record.
Why it wasn't detected
Each mode was internally consistent, so a check within any single mode would pass:
- Native streaming: IDs and records were missing at the same positions, so they agreed with each other.
- Non-streaming: IDs were complete but records were short, so every record after the first gap was attached to the wrong token.
The fix
PR #31244 fixes both defects in two commits:
- Queue token IDs. A per-slot queue takes every sampled token's ID immediately and sends the whole queue with the next released chunk, or at the end of the stream. With IDs complete, a chunk could carry several IDs but still only one probability record, which exposed the second defect.
- Queue probability records the same way, including chat deltas that carry no text.
What MURPHY measures
MURPHY is a 1,500-prompt benchmark that checks whether an inference server reports every token ID and probability record correctly on text where characters span several tokens. It scores record integrity separately from answer quality.
| Dataset | Prompts | Task |
|---|---|---|
| FLORES+ (dev) | 900 | Copy a sentence exactly, in English, Russian, Spanish, Modern Standard Arabic, Japanese, or Traditional Chinese |
| Belebele | 300 | Answer a reading question with the letter and full text of the chosen option, same six languages |
| Unicode Emoji | 300 | Copy 10 emoji-and-name lines exactly, balanced across single code points, modifiers, ZWJ sequences, flags, and other sequences |
Test setup
Every prompt ran in every combination below, for 24,000 requests in total (1,500 × 2 × 2 × 2 × 2).
| Factor | Values |
|---|---|
| Build | Baseline, patched |
| Endpoint | Native /completion, chat /v1/chat/completions |
| Mode | Streaming, non-streaming |
| Repeats | 2 (confirm stability; they don't double the evidence) |
Held fixed on both builds: Qwen3.5-9B BF16, CUDA with flash attention off, temperature 0, seed 42, concurrency 1, n_probs: 5.
Checks
| Check | Compares | Scope |
|---|---|---|
| Missing records | Records vs. token IDs, and on chat, vs. usage.completion_tokens |
All |
| Token identity | Streamed token sequence vs. the same build's non-streaming reference | Native only |
| Byte labels | Each record's bytes vs. that token's bytes from an independent GGUF vocabulary decoder | All |
| Numerical agreement | Logprobs and top candidates across builds (tolerance 1e-6 absolute, 1e-5 relative) | Only where both histories align |
| Instruction following | Strict exact-match compliance, kept separate so a fix that changed answers would show up | All |
Results
The patched build had zero record defects in all 12,000 of its responses, and instruction-following scores did not change.
Record integrity
Responses with at least one record defect, out of all responses in each cell (both repeats). These are reporting failures, not wrong answers.
On emoji, the baseline lost about a quarter of its records:
- Native and chat non-streaming each omitted 24.9% of probability records.
- Native streaming omitted the same 24.9% of token IDs and records together, so its own counts matched, but 100% of responses still failed the identity check.
- Chat streaming omitted 25.6% of records: the same 24.9%, plus exactly one more per response.
- Every endpoint and mode had wrong selected-token byte labels on 14.0% of records.
- The patched build reported 100% of records in every endpoint and mode.
Instruction following
Streaming and non-streaming scored identically, and the patch changed nothing.
| Dataset | Native | Chat |
|---|---|---|
| FLORES+ | 92.78% (1,670 / 1,800) | 92.78% (1,670 / 1,800) |
| Belebele | 81.33% (488 / 600) | 83.67% (502 / 600) |
| Unicode Emoji | 84.67% (508 / 600) | 84.67% (508 / 600) |
Belebele's higher chat score comes entirely from seven English prompts whose chat answers lacked an unwanted leading space (14 responses per mode across both repeats). The chosen answers were the same.
How long has this existed?
The loss dates to October 22, 2023, about three years before the fix. Later commits added fields and endpoints that inherited the same buffering defect, so there is no single introducing commit.
- How it was dated:
git bisectover 945 commits found the 2023 commit in ten steps with no skips, and its parent tested good. The test model (stories260K-f32.gguf) encodes each emoji byte as one token, so the expected record count is exact: forA🙂B, the last good commit returned 6 records and every defective one returned 3. Later commits were each tested against their parent. - Not covered: probability bugs before October 2023, and losses from stop-string trimming, mid-character truncation, reasoning templates, or speculative decoding.
Limitations and open questions
These results hold for one model and one configuration. What they don't establish:
- Other models and tokenizers. Qwen3.5's vocabulary is about 248K tokens, near the top of the range. Models with smaller vocabularies, such as Llama 2 or the original Mistral 7B at 32K, likely split more characters, so these numbers are probably closer to a lower bound.
- Other setups: quantizations, backends, flash attention, concurrency above 1, and other chat or reasoning templates.
- Chat token identity. Chat exposes no independent token history, so on chat, zero failures means the count and byte checks passed, not that token identity or numerical agreement was proven.
/v1/completionsat scale. It was covered by the bisect's fresh tests, where it failed and was fixed exactly like/completion, but not by the 24,000-request MURPHY run.- Numerical accuracy. No independent floating-point reference was used, so neither MURPHY nor the bisect claims anything about logprob values being wrong, only missing or mislabeled.
Open questions for follow-up:
- Does partial stop-string matching, which also holds text back, cause the same loss on plain ASCII? Stop-word trimming dates to September 2023, as an intended text policy, so its effect on records needs its own test.
- Do speculative decoding and MTP, which accept several tokens per step, go through the fixed path?
- What should be reported for tokens that form a matched stop string, whose text is removed from the output?
- Do tool-call and reasoning parsing on the chat path hold back text in a way the fix doesn't cover?
- How much of real data, like agent trajectories with terminal output, was affected?
Reproducibility
The evaluated dataset, measurement source and evidence are publicly available at the pinned revisions below.
| Component | Value |
|---|---|
| Baseline llama.cpp | 8a199ad31a7eb27d5a46a6b08026176d82bc9894 |
| Patched llama.cpp | 636d96d2e686690011efc1f6a265692c380ab44b (PR #31244) |
| Model | Qwen3.5-9B BF16 GGUF |
| Model SHA-256 | 30a6732abca868d08fe0e9abc9a69d0b2c8653cd9ad07e60e8ea675fef06a1be |
| Evaluated MURPHY source | 9557ccdfd71acd2bd47d125b24ca43e559b49f04; included in the evaluation archive |
| Public MURPHY code | bytkim/MURPHY-BENCH, commit 3864cb24e3b9ac2620619901a17efd6c429f2e02 |
| Dataset and evidence | bytkim/MURPHY-BENCHMARK, revision c930bbd09898797c3a2b8206859866d596be56f9 |
| Hardware | One NVIDIA RTX 4090, CUDA, driver 595.91.07; flash attention off |
| Runtime settings | Context 4096; batch/microbatch 512; GPU layers 99; one thread and slot; backend sampling off |
| Generation settings | Temperature 0; seed 42; five probabilities; maximum 384 generated tokens; thinking off; no grammar |
The dataset folder contains the exact 1,500 evaluated tasks (900 FLORES+, 300 Belebele and 300 Unicode Emoji prompts) with their original build manifests. These files were extracted byte-for-byte from the preserved evaluation, not regenerated.
The evaluation archive contains all 24,000 saved responses, grades, comparisons, measurement code and relevant logs. The endpoint-separated report is also available separately.
Acknowledgements
Thanks to the llama.cpp maintainers and reviewers for their time and feedback on PR #31244. Thanks also to the creators of FLORES+, Belebele and the Unicode Emoji data, which MURPHY is built on.
Patches and evaluation tooling were written with AI coding assistance; I designed the experiments, reviewed the code, and verified the results.



