whoisjiji/verifiability-classifier

Verifiability Classifier

The Verifiability Classifier is a 0.6B-parameter binary text classifier that estimates whether the answer to a question can be checked reliably with a compact verifier. It is intended for selecting instruction and reasoning tasks whose outcomes support deterministic, parser-based, or regular-expression rewards.

The model is introduced in Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning.

Typical positive examples include numerical calculations, multiple-choice questions, and other tasks with short answers that can be normalized and compared objectively. Open-ended analysis, recommendations, and explanations without a compact correctness criterion are normally negative examples.

The model predicts verifiability, not correctness: a question may be classified as verifiable even when a supplied answer is wrong.

Model

The checkpoint adds a two-class sequence-classification head to Qwen/Qwen3-Embedding-0.6B. It consumes the user question, truncated to at most 4,096 tokens.

Label	Meaning
NOT_VERIFIABLE	Correctness cannot be evaluated reliably with a compact verifier.
VERIFIABLE	The answer has a compact, objectively checkable outcome.

Usage

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="whoisjiji/verifiability-classifier",
    tokenizer="whoisjiji/verifiability-classifier",
)

questions = [
    "A bond pays a $50 annual coupon and costs $950. What is its current yield?",
    "Discuss whether active investing is better than passive investing.",
]

scores = classifier(
    questions,
    top_k=None,
    truncation=True,
    max_length=4096,
)

for question, result in zip(questions, scores):
    probabilities = {item["label"]: item["score"] for item in result}
    print(question, probabilities["VERIFIABLE"])

The output is a selection score rather than a universal decision boundary. Calibrate the threshold on manually reviewed examples from the intended data source. Higher thresholds are appropriate when false positives would introduce noisy reinforcement-learning rewards.

Limitations

• The classifier does not generate a verifier and does not prove that a reliable verifier has been implemented.
• Some apparently numerical tasks still require subjective assumptions or unavailable data.
• Some structured tasks may be objectively evaluable even when their expected answer is not short.
• The model was trained primarily for English financial and synthetic instruction data; performance may differ across languages and domains.
• This model is intended for dataset curation and research, not financial advice or automated high-stakes decisions.

Citation

@article{hayrapetyan2026datacentric,
  title   = {Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning},
  author  = {Hayrapetyan, Zhirayr and Kalmykov, Andrei and Kokosinskii, Denis and Stanishevskii, Dmitry and Zmitrovich, Dmitry},
  journal = {arXiv preprint arXiv:2609.10113},
  year    = {2026}
}

@misc{hayrapetyan2026verifiability,
  title  = {Verifiability Classifier},
  author = {Hayrapetyan, Zhirayr},
  year   = {2026},
  url    = {https://huggingface.co/whoisjiji/verifiability-classifier}
}