KRLabsOrg/LFM2.5-Encoder-350M-hallucination-detector

LFM2.5-Encoder-350M hallucination detector

A span-level hallucination detector built on LiquidAI/LFM2.5-Encoder-350M: the bidirectional LFM2.5 backbone with a linear token-classification head, fine-tuned to mark the character spans of an answer that are not supported by the given context. Part of the LettuceDetect project.

Trained on the LettuceDetect unified benchmark: coding-agent answers (SWE-bench-derived), developer tool output, structured documents (ACL papers, READMEs, Wikipedia markdown), RAGTruth, and 14-language PsiloQA.

Results

Character-level span metrics on the LettuceDetect unified test split (n=10,698):

group	n	span-F1	span-P	span-R	example-F1	IoU
ALL	10698	0.603	0.669	0.549	0.854	0.620
lettucedetect-acl	440	0.488	0.671	0.383	0.798	0.529
lettucedetect-code-agent	2015	0.443	0.582	0.357	0.744	0.496
lettucedetect-readme	641	0.711	0.763	0.665	0.879	0.727
lettucedetect-tool-output	617	0.525	0.688	0.424	0.719	0.576
lettucedetect-wikipedia	1388	0.668	0.729	0.617	0.875	0.704
psiloqa (14 languages)	2897	0.690	0.684	0.696	0.945	0.588
ragtruth	2700	0.463	0.696	0.347	0.744	0.702

Comparison

Span-F1 by source against the other LettuceDetect detectors on the same test split:

source	this (350M)	mmbert-base (307M)	qwen-2b (2B)
ALL	0.603	0.642	0.689
acl	0.488	0.579	0.749
code-agent	0.443	0.508	0.602
readme	0.711	0.751	0.866
tool-output	0.525	0.588	0.719
wikipedia	0.668	0.708	0.817
psiloqa (14 languages)	0.690	0.714	0.732
ragtruth	0.463	0.528	0.574

On multilingual example-level detection it matches the mmBERT encoder (PsiloQA example-F1 0.945 vs 0.943). On code-agent answers it remains far above general-purpose LLM judges at a fraction of their size (Nemotron-3-Ultra-550B 0.216, gpt-oss-120b 0.212 span-F1; HHEM-2.1 / Lynx-8B / Granite-Guardian / MiniCheck ≈ chance).

Usage

# pip install lettucedetect
from lettucedetect.models.inference import HallucinationDetector

detector = HallucinationDetector(
    method="transformer",
    model_path="KRLabsOrg/LFM2.5-Encoder-350M-hallucination-detector",
    trust_remote_code=True,
)

predictions = detector.predict(
    context=["The Eiffel Tower is 330 metres tall and stands in Paris, France."],
    question="How tall is the Eiffel Tower and where is it?",
    answer="The Eiffel Tower is 330 metres tall and stands in Berlin.",
    output_format="spans",
)
print(predictions)
# [{'start': 49, 'end': 56, 'confidence': 0.92, 'text': ' Berlin'}]

Token-level classification without the LettuceDetect wrapper:

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "KRLabsOrg/LFM2.5-Encoder-350M-hallucination-detector"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo, trust_remote_code=True).eval()

enc = tokenizer("context text ... answer text", return_tensors="pt")
with torch.no_grad():
    labels = model(**enc).logits.argmax(-1)[0]   # 0 = supported, 1 = hallucinated

Training

• Backbone: LiquidAI/LFM2.5-Encoder-350M (bidirectional), linear head, dropout 0.1
• 3 epochs on the unified train split (66,368 samples), input [question, context, answer], answer tokens labeled supported/hallucinated, max length 8,192
• Token-level validation F1 0.635, test F1 0.609

Citation

@misc{kovács2026documentgroundingspanlevelhallucination,
      title={Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents},
      author={Ádám Kovács and Bowei He and Xue Liu and István Boros and Szilveszter Tóth and Gábor Recski},
      year={2026},
      eprint={2607.00895},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.00895},
}