Fixing Silent Token-ID and Logprob Loss in llama.cpp Server, Measured with MURPHY

Community Article
Published October 31, 2026

TL;DR

llama.cpp's server silently dropped or mislabeled token IDs and logprobs whenever one character was split across several tokens. The generated text was always correct, so nothing looked wrong.

  • What broke: the server holds back text until a UTF-8 character is complete. Token IDs and probability records were attached per released text chunk instead of per sampled token, so tokens inside a chunk lost their IDs, their logprobs, or both.
  • How much: on emoji-heavy output with Qwen3.5-9B, every unpatched response was affected, and 22,500 of 90,380 probability records went missing per endpoint and mode.
  • Where: native /completion, /v1/completions, and /v1/chat/completions, in both streaming and non-streaming modes.
  • Since when: probability records since October 2023, about three years; streamed token IDs since token output was added in December 2024.
  • Fixed by: PR #31244, in two commits: one for token IDs, one for probability records. On the patched build, MURPHY found zero record defects across 24,000 requests, with unchanged instruction-following scores.

If you collected token IDs or logprobs from llama-server, especially on non-English text, emoji, or chat streaming, check your data. The next section shows how.

Should you check your data?

Your data may be affected if all of these hold:

  1. It came from llama-server, on a build before the fixes.
  2. You kept token IDs (tokens on native /completion, populated whenever return_tokens or stream is true) or logprobs (n_probs, logprobs, top_logprobs).
  3. The output contained characters split across several tokens: non-English scripts, emoji, symbols, smart quotes, arrows, or box-drawing characters from terminal output. Chat streaming was affected even on English text.

What it affects depends on how you used the data:

Use Affected?
SFT on text, retokenized from the output ✅ No: the text was correct
Pass/fail evals scored on output text ✅ No
RL reusing stored logprobs (importance ratios, KL) ❌ Yes: misaligned logprobs corrupt the training signal
Distillation from top-k candidates ❌ Yes: candidate records and byte labels were affected
Calibration, uncertainty, or perplexity from logprobs ❌ Yes: biased toward non-English and emoji-heavy text
Pipelines that rebuild text from token IDs ❌ Yes: dropped IDs corrupt the text itself

Three cheap checks, no rerun needed:

  1. Count check: the number of logprob records should equal the number of token IDs, and on chat, usage.completion_tokens.
  2. Byte check: decode each record's token ID with the model's tokenizer and compare with that record's bytes. Don't just concatenate the bytes and compare with the text: the bug moves a whole character's bytes onto one record, so the concatenation still matches.
  3. Reference check: for a sample, rerun the prompt non-streaming at temperature 0 on a patched build and compare token by token.

How the bug works

The server reported each token only when it released that token's text, so tokens inside a split character lost their reports.

Root cause

The server sends only whole UTF-8 characters. When a character spans several tokens, it holds the text back until the last token arrives. Token reporting was tied to that text release, so it followed the text, not the tokens.

Two defects

Token IDs Probability records
What went wrong Held-back tokens' IDs were dropped Saved only after text processing: held-back tokens got no record, and the surviving record carried the whole character's bytes
Where Native streaming only; non-streaming kept every ID Every mode, on both endpoints

Worked example: A🙂B

Take A🙂B on the bisect's test model, whose vocabulary encodes each of the emoji's four UTF-8 bytes as its own token: six tokens in all. Before the fix, non-streaming returned all six IDs but only three records, and streaming returned three of each.

Sampled token Bytes Text released? Baseline token ID (streaming) Baseline logprob record Patched
A 41 Yes Reported Reported Both reported
🙂 byte 1 F0 No, held Dropped Dropped Both queued
🙂 byte 2 9F No, held Dropped Dropped Both queued
🙂 byte 3 99 No, held Dropped Dropped Both queued
🙂 byte 4 82 Yes, as 🙂 Reported Reported, labeled F0 9F 99 82 All four sent with 🙂
B 42 Yes Reported Reported Both reported

Qwen3.5 splits characters differently, but the mechanism is the same: any character that spans several tokens loses all but its last token's record.

Why it wasn't detected

Each mode was internally consistent, so a check within any single mode would pass:

  • Native streaming: IDs and records were missing at the same positions, so they agreed with each other.
  • Non-streaming: IDs were complete but records were short, so every record after the first gap was attached to the wrong token.

The fix

PR #31244 fixes both defects in two commits:

  1. Queue token IDs. A per-slot queue takes every sampled token's ID immediately and sends the whole queue with the next released chunk, or at the end of the stream. With IDs complete, a chunk could carry several IDs but still only one probability record, which exposed the second defect.
  2. Queue probability records the same way, including chat deltas that carry no text.

What MURPHY measures

MURPHY is a 1,500-prompt benchmark that checks whether an inference server reports every token ID and probability record correctly on text where characters span several tokens. It scores record integrity separately from answer quality.

Dataset Prompts Task
FLORES+ (dev) 900 Copy a sentence exactly, in English, Russian, Spanish, Modern Standard Arabic, Japanese, or Traditional Chinese
Belebele 300 Answer a reading question with the letter and full text of the chosen option, same six languages
Unicode Emoji 300 Copy 10 emoji-and-name lines exactly, balanced across single code points, modifiers, ZWJ sequences, flags, and other sequences

Test setup

Every prompt ran in every combination below, for 24,000 requests in total (1,500 × 2 × 2 × 2 × 2).

Factor Values
Build Baseline, patched
Endpoint Native /completion, chat /v1/chat/completions
Mode Streaming, non-streaming
Repeats 2 (confirm stability; they don't double the evidence)

Held fixed on both builds: Qwen3.5-9B BF16, CUDA with flash attention off, temperature 0, seed 42, concurrency 1, n_probs: 5.

Checks

Check Compares Scope
Missing records Records vs. token IDs, and on chat, vs. usage.completion_tokens All
Token identity Streamed token sequence vs. the same build's non-streaming reference Native only
Byte labels Each record's bytes vs. that token's bytes from an independent GGUF vocabulary decoder All
Numerical agreement Logprobs and top candidates across builds (tolerance 1e-6 absolute, 1e-5 relative) Only where both histories align
Instruction following Strict exact-match compliance, kept separate so a fix that changed answers would show up All

Results

The patched build had zero record defects in all 12,000 of its responses, and instruction-following scores did not change.

Record integrity

Responses with at least one record defect, out of all responses in each cell (both repeats). These are reporting failures, not wrong answers.

Heatmap of responses with record defects: native and chat non-streaming and native streaming had 4.3% on FLORES+ and 0.7% on Belebele; chat streaming had 100% on both; every baseline mode had 100% on Unicode Emoji; the patched build had 0% everywhere.

On emoji, the baseline lost about a quarter of its records:

  • Native and chat non-streaming each omitted 24.9% of probability records.
  • Native streaming omitted the same 24.9% of token IDs and records together, so its own counts matched, but 100% of responses still failed the identity check.
  • Chat streaming omitted 25.6% of records: the same 24.9%, plus exactly one more per response.
  • Every endpoint and mode had wrong selected-token byte labels on 14.0% of records.
  • The patched build reported 100% of records in every endpoint and mode.

Instruction following

Streaming and non-streaming scored identically, and the patch changed nothing.

Dataset Native Chat
FLORES+ 92.78% (1,670 / 1,800) 92.78% (1,670 / 1,800)
Belebele 81.33% (488 / 600) 83.67% (502 / 600)
Unicode Emoji 84.67% (508 / 600) 84.67% (508 / 600)

Belebele's higher chat score comes entirely from seven English prompts whose chat answers lacked an unwanted leading space (14 responses per mode across both repeats). The chosen answers were the same.

How long has this existed?

The loss dates to October 22, 2023, about three years before the fix. Later commits added fields and endpoints that inherited the same buffering defect, so there is no single introducing commit.

Vertical timeline from Oct 22, 2023 (438c2ca83045, the origin) through seven commits that inherited the defect, ending Dec 31, 2024, then 22 months with no error or warning until the fix, PR #31244, on Oct 27, 2026.

  • How it was dated: git bisect over 945 commits found the 2023 commit in ten steps with no skips, and its parent tested good. The test model (stories260K-f32.gguf) encodes each emoji byte as one token, so the expected record count is exact: for A🙂B, the last good commit returned 6 records and every defective one returned 3. Later commits were each tested against their parent.
  • Not covered: probability bugs before October 2023, and losses from stop-string trimming, mid-character truncation, reasoning templates, or speculative decoding.

Limitations and open questions

These results hold for one model and one configuration. What they don't establish:

  • Other models and tokenizers. Qwen3.5's vocabulary is about 248K tokens, near the top of the range. Models with smaller vocabularies, such as Llama 2 or the original Mistral 7B at 32K, likely split more characters, so these numbers are probably closer to a lower bound.
  • Other setups: quantizations, backends, flash attention, concurrency above 1, and other chat or reasoning templates.
  • Chat token identity. Chat exposes no independent token history, so on chat, zero failures means the count and byte checks passed, not that token identity or numerical agreement was proven.
  • /v1/completions at scale. It was covered by the bisect's fresh tests, where it failed and was fixed exactly like /completion, but not by the 24,000-request MURPHY run.
  • Numerical accuracy. No independent floating-point reference was used, so neither MURPHY nor the bisect claims anything about logprob values being wrong, only missing or mislabeled.

Open questions for follow-up:

  • Does partial stop-string matching, which also holds text back, cause the same loss on plain ASCII? Stop-word trimming dates to September 2023, as an intended text policy, so its effect on records needs its own test.
  • Do speculative decoding and MTP, which accept several tokens per step, go through the fixed path?
  • What should be reported for tokens that form a matched stop string, whose text is removed from the output?
  • Do tool-call and reasoning parsing on the chat path hold back text in a way the fix doesn't cover?
  • How much of real data, like agent trajectories with terminal output, was affected?

Reproducibility

The evaluated dataset, measurement source and evidence are publicly available at the pinned revisions below.

Component Value
Baseline llama.cpp 8a199ad31a7eb27d5a46a6b08026176d82bc9894
Patched llama.cpp 636d96d2e686690011efc1f6a265692c380ab44b (PR #31244)
Model Qwen3.5-9B BF16 GGUF
Model SHA-256 30a6732abca868d08fe0e9abc9a69d0b2c8653cd9ad07e60e8ea675fef06a1be
Evaluated MURPHY source 9557ccdfd71acd2bd47d125b24ca43e559b49f04; included in the evaluation archive
Public MURPHY code bytkim/MURPHY-BENCH, commit 3864cb24e3b9ac2620619901a17efd6c429f2e02
Dataset and evidence bytkim/MURPHY-BENCHMARK, revision c930bbd09898797c3a2b8206859866d596be56f9
Hardware One NVIDIA RTX 4090, CUDA, driver 595.91.07; flash attention off
Runtime settings Context 4096; batch/microbatch 512; GPU layers 99; one thread and slot; backend sampling off
Generation settings Temperature 0; seed 42; five probabilities; maximum 384 generated tokens; thinking off; no grammar

The dataset folder contains the exact 1,500 evaluated tasks (900 FLORES+, 300 Belebele and 300 Unicode Emoji prompts) with their original build manifests. These files were extracted byte-for-byte from the preserved evaluation, not regenerated.

The evaluation archive contains all 24,000 saved responses, grades, comparisons, measurement code and relevant logs. The endpoint-separated report is also available separately.

Acknowledgements

Thanks to the llama.cpp maintainers and reviewers for their time and feedback on PR #31244. Thanks also to the creators of FLORES+, Belebele and the Unicode Emoji data, which MURPHY is built on.

Patches and evaluation tooling were written with AI coding assistance; I designed the experiments, reviewed the code, and verified the results.

Community

Sign up or log in to comment