yah01/vjev-vision-pilot

vjev — vision pilot (step 300)

A listwise decision model with vision: give it a state (text, images, or both) and typed questions — noul (is this statement true?), choice (pick one), score (an ordered scale) — and it returns calibrated probabilities for every option, in a single forward pass, with no text generation. It is a re-creation of the Jev API's shape on an open base, with images added.

This is a pilot checkpoint (300 steps of vision training), not a finished model. It is the checkpoint served by the project's web console while the full run trains.

Use it

The web console and the HTTP API live in BubbleCal/vjev-serve:

pip install git+https://github.com/BubbleCal/vjev-serve
vjev-serve --model yah01/vjev-vision-pilot        # then open http://localhost:8800

Or, with nothing but transformers, the single file in this repo:

pip install "transformers>=5.15" torch pillow safetensors huggingface_hub

from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vjev_infer", hf_hub_download("yah01/vjev-vision-pilot", "vjev_infer.py"))
vjev_infer = importlib.util.module_from_spec(spec); spec.loader.exec_module(vjev_infer)

m = vjev_infer.Vjev("yah01/vjev-vision-pilot")        # cuda / mps / cpu
m.ask(state=["photo.jpg", "Frame from the warehouse camera, 12:40."],
      questions={"person": {"type": "noul",   "instructions": "There is a person in this image."},
                 "where":  {"type": "choice", "instructions": "Where is the forklift?",
                            "criteria": {"left": "left half", "right": "right half", "none": "no forklift"}},
                 "busy":   {"type": "score",  "instructions": "How cluttered is the scene?",
                            "criteria": ["empty", "sparse", "busy", "crowded"]}})

Every question is answered with a probability distribution over its options (or one probability for a noul statement), from one forward pass; nothing is generated. Options inside one question compete (they share one softmax); questions never see each other. To rate several independent things, ask one noul per thing.

vjev_infer.py (~170 lines) is the whole inference path: the prompt template, the listwise head, image resizing. Needs ~9 GB of memory in bf16/fp16.

What is in this repo

file	what
model-*.safetensors, config.json, tokenizer and processor files	the full model, bf16: Qwen3.5-4B with this checkpoint's LoRA merged in
head.pt	the listwise scoring head: one shared linear layer read at each option's slot
vjev.json	how inputs are rendered for this checkpoint (readout: trailing, pause: 0, length budgets)
vjev_infer.py	stand-alone inference (above); the full server and console are in BubbleCal/vjev-serve
adapter/	the same weights as a LoRA adapter (PEFT, r=32, α=64) over Qwen/Qwen3.5-4B — 248 language-model modules and 98 vision-tower modules

The merged weights are the nf4-dequantized base the adapter was trained against, plus the adapter, in bf16. Checked against the adapter served on the original base: probabilities agree to within 0.001. The model's lm_head is present but unused.

How it was trained

1. Text stage (text_v2, 3,868 steps): QLoRA (nf4 base, bf16 compute) on ~145k typed questions whose soft labels came from the official Jev API, plus human-labelled corpora (banking77, clinc_oos, go_emotions, ...). Loss: KL to the teacher distribution for choice/score, soft BCE for noul. Every option of a question lives in one sequence and is read at a trailing Answer: (A) (B) ... slot, so options can see each other (a pointwise scorer cannot reproduce Jev's option interactions).
2. Vision stage (this checkpoint, 300 steps, warm-started from the text stage): a second LoRA on the vision tower (at 0.1× the text learning rate), on ~84k geometry questions derived from COCO-2017 annotations (which object is highest / smallest / left of ..., counts, presence with adversarial absent objects; soft labels by Monte-Carlo perturbation of the boxes) and ~15k VQAv2 questions with their 10-annotator answer distributions, mixed with 25% text.

Results (held-out images, none seen in training)

split	metric	text stage (zero-shot on images)	this checkpoint
spatial_val (COCO geometry)	choice accuracy	0.506	0.665
spatial_val	choice ECE (lower is better)	0.171	0.094
vqa_val	choice accuracy	0.579	0.654
vqa_val	choice ECE	0.126	0.033
POPE, adversarial	agreement (yes/no)	0.875	0.856

Known weakness: yes/no judgements about object presence drifted towards "yes" during the pilot — on POPE's adversarial set, absent objects are called present 13.8% of the time (text stage: 5.5%). The full run monitors this per checkpoint.

On text, the text stage matches its teacher's accuracy on 20k held-out rows (0.796 vs 0.797 against human labels) with better calibration (ECE 0.068 vs 0.102).

Loading the adapter instead

adapter/ was fitted on the nf4-quantized base. Under it, load Qwen/Qwen3.5-4B the same way (bitsandbytes, load_in_4bit, nf4, double quant, bf16 compute): the plain bf16 base runs without error and scores a slightly different model. The merged weights above already include the dequantized base, so they need no bitsandbytes.

Limits

• Pilot: 300 steps. Spatial questions of the harder families (size_smallest, extreme_top) are at 0.60–0.62 accuracy.
• Trained on single images only; several images in one request (Picture 1:, Picture 2:) work through the base model's ability, unverified beyond a handful of synthetic checks.
• License: Apache-2.0 (the base, Qwen3.5-4B, is Apache-2.0).
• Options are read in the order given; reordering them moves probabilities by a TVD of ~0.035 on average (the teacher API shows ~0.048).