FastLFM2.5-VL-0.5B

FastLFM2.5-VL-0.5B retains 92% of its teacher's average captioning quality while delivering 1.5x throughput and using 4x fewer visual tokens.

This model combines the vision encoder from Apple FastVLM-0.5B with the language model from LiquidAI LFM2.5-VL-450M. A small residual connector maps the FastVLM image features into the LFM2 embedding space.

The Apple vision encoder and Liquid language model remained frozen. Only the connector path, consisting of the residual MLP and its learned 2D positional embeddings, was trained. Training used supervised fine-tuning (SFT) followed by on-policy distillation (OPD) from the LFM2.5-VL-450M teacher.

Results

Model	Input size	COCO CIDEr	COCO retention	NoCaps CIDEr	NoCaps retention	Visual tokens	Throughput vs. teacher
LiquidAI/LFM2.5-VL-450M	512x512	0.9109	100%	0.9443	100%	256 (1x)	1.00x
apple/FastVLM-0.5B	512x512	0.7606	83.5%	0.8103	85.8%	64 (4x fewer)	1.53x
FastLFM2.5-VL-0.5B	512x512	0.8309	91.2%	0.8782	93.0%	64 (4x fewer)	1.51x

The two FastLFM retention scores average to 92.1%. Retention is computed independently for each benchmark against the 512x512 LFM teacher.

Throughput was measured on 1,000 unique, already-loaded 512x512 images on an NVIDIA RTX PRO 6000 Blackwell Server Edition, using batches of 250 and exactly 32 generated tokens after three warmup batches.

Architecture

512x512 image; 64 visual tokens; d_vision=3072; d_LLM=1024

                    ┌─────────────────────┐
                    │ FastVLM Encoder     │  frozen
                    └──────────┬──────────┘
                               │ [B, 64, 3072]
              ┌────────────────┴────────────────┐
              │                                 │
    ┌─────────┴──────────┐           ┌──────────┴─────────┐
    │ Skip Linear        │           │ Standardize        │
    │ 3072 → 1024        │           │ Linear 3072 → 3072 │
    └─────────┬──────────┘           │ ReLU²              │
              │                      │ Linear 3072 → 1024 │
              │                      │ Scale × alpha      │
              │                      └──────────┬─────────┘
              └────────────────┬────────────────┘
                               │ residual addition
                    ┌──────────┴──────────┐
                    │ Learned 8x8 2D      │
                    │ positions           │
                    └──────────┬──────────┘
                               │ [B, 64, 1024]
                    ┌──────────┴──────────┐
                    │ LFM2 Language Model │  frozen
                    └─────────────────────┘

Component	Parameters	Training status
Apple FastVLM-derived vision encoder	125,073,504 (125.1M)	Frozen
Residual connector MLP and learned 8x8 2D positions	15,750,145 (15.75M)	Trained: SFT, then OPD
Liquid LFM2 language model	354,483,968 (354.5M)	Frozen
Total	495,307,617 (495.31M)	15.75M trainable (3.18%)

Maximum-throughput vLLM

For large offline captioning jobs, use the included split vLLM path. It compiles the vision encoder, processes 250 images together, and passes the resulting 64-token prompt embeddings to vLLM's native LFM2 decoder. No patched or custom vLLM installation is required.

pip install "vllm==0.25.1" "transformers==5.13.1" \
  huggingface_hub safetensors pillow numpy

hf download irotem98/FastLFM2.5-VL-0.5B \
  --local-dir FastLFM2.5-VL-0.5B

python FastLFM2.5-VL-0.5B/fast_vllm.py \
  --model FastLFM2.5-VL-0.5B \
  --image-dir ./images \
  --output captions.jsonl \
  --batch-size 250 \
  --max-tokens 128 \
  --gpu-memory-utilization 0.85

Training

Training updated only the 15.75M-parameter connector path:

1. SFT: supervised alignment of the FastVLM visual features to the frozen LFM2 language model.
2. OPD: on-policy distillation from the LFM2.5-VL-450M teacher, refining the connector while keeping both foundation components frozen.

Intended use and limitations

This checkpoint is optimized for fast, one-sentence English image descriptions at 512x512 resolution. It always emits an 8x8 grid of 64 visual tokens. Quality and throughput figures above are captioning results under the stated setup, not guarantees for other prompts, resolutions, GPUs, batch sizes, or generation lengths.

License and attribution

This repository combines components derived from Apple FastVLM-0.5B and LiquidAI LFM2.5-VL-450M. Review and preserve both upstream licenses and usage terms before redistribution. The bundled FastVLM implementation retains its upstream Apache-2.0 source header; this model card does not replace either upstream license.