LFM2.5-1.2B-Instruct — LiteRT-LM

LiquidAI/LFM2.5-1.2B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14 / a recent AI Edge Gallery).

Update (2026-08-04): the .litertlm files were updated in place to add the ExecutorMetadata section that litert-lm ≥ 0.15 requires to bind the hybrid conv/attention state buffers (without it, 0.15 fails at inference with missing some output TensorBuffers). Weights and graph are byte-identical to the original release, and the files continue to run on litert-lm 0.14.

LFM2.5-1.2B-Instruct is Liquid AI's flagship on-device model: a hybrid architecture mixing gated short-convolution blocks with grouped-query attention. Per the base model card it outperforms Qwen3-1.7B and Gemma 3 1B on knowledge and instruction-following benchmarks at a smaller footprint.

File	Recipe	Size	GSM8K (n=100)
LFM2.5-1.2B-Instruct_int8.litertlm	int8 dynamic (linears + convs + embedding)	1.25 GB	81% (bf16 reference: 79%)
LFM2.5-1.2B-Instruct_int4.litertlm	int4 blockwise-32 + OCTAV linears, int8 embedding, convs float	736 MB	72%
LFM2.5-1.2B-Instruct_int4_gpu.litertlm	same int4 recipe, re-exported so it runs on the GPU	736 MB	= int4 (A/B in notes)

Context (KV cache)	4096 max (see --max-num-tokens note)
Backend	CPU for every file; the _int4_gpu file also runs on the GPU with litert-lm ≥ 0.16.0 (Android OpenCL and macOS — verified by generation, not just by a benchmark table; iOS Metal still fails at engine creation, tracked upstream in LiteRT-LM#3129)
Template	bundled — full chat template incl. tool-list support + <think> channel declaration
Base model	LiquidAI/LFM2.5-1.2B-Instruct (LFM Open License v1.0)

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 1024, n=100, same harness for all rows:

Configuration	GSM8K
PyTorch bf16 (reference)	79%
LiteRT int8 (this repo)	81%
LiteRT int4-b32 OCTAV (this repo)	72%

The int8 file is at full parity with the PyTorch reference (+2pt, within noise). Both files pass an 8-question sanity gate (7/8, zero degenerate outputs). The int4 file trades −7pt for a 41% smaller download.

Usage

litert-lm run ./LFM2.5-1.2B-Instruct_int8.litertlm --prompt "Explain what a hybrid conv-attention LLM is in two sentences."

The bundle carries the tokenizer and the full chat template (ChatML-style <|im_start|>…<|im_end|> with tool-calling support). Generation stops cleanly at <|im_end|>.

--max-num-tokens tip: decode speed drops as the token budget grows (int8 decode: 101 tok/s at 1024 → 77 tok/s at 4096 on an M4 Max) — set it to the smallest value your use case needs (1024 is a good chat default; the file allows up to 4096).

Performance

litert-lm benchmark (litert-lm 0.15.0), CPU backend, Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), warm-up run discarded, otherwise idle machine: decode depends strongly on the KV budget, so both settings are listed:

Variant	--max-num-tokens	Prefill (256)	Decode	TTFT
int8	1024	1592 tok/s	93.1 tok/s	0.17 s
int4	1024	381 tok/s	109.4 tok/s	0.68 s
int8	4096	1121 tok/s	78.9 tok/s	0.24 s
int4	4096	343 tok/s	74.9 tok/s	0.76 s

On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery): int8 decodes at ~19 tok/s and int4 at ~31 tok/s — on phone-class memory bandwidth the int4 file is ~1.7× faster as well as 41% smaller, so prefer int4 on mid-range devices unless you need the last few GSM8K points.

GPU — the _int4_gpu file

Pixel 8a (Tensor G3), litert_lm_main built from the litert-lm v0.16.0 release tag, 263-token prompt, generation capped at 256, 3 runs per backend:

Backend	Prefill (263 tok)	Decode	TTFT
GPU (OpenCL)	188–193 tok/s	21.0–21.2 tok/s	1.41–1.44 s
CPU	38–54 tok/s	15.2–24.3 tok/s	4.9–7.0 s

On phone-class hardware the GPU's win is prefill and time-to-first-token (3–6× both); decode is bandwidth-bound and roughly a wash, so long prompts gain far more than long answers. The CPU rows spread because the phone throttles across a run series; the GPU rows repeat to within 2%.

Apple M4 Max (litert-lm benchmark --cache no, litert-lm 0.16.0, -p 256 -d 256 --runs 3, otherwise idle machine; both backends were verified to generate correct text before these numbers were quoted):

Backend	Prefill (256)	Decode	TTFT
GPU	3765 tok/s	318.3 tok/s	0.07 s
CPU	337.0 tok/s	80.5 tok/s	0.78 s

On a desktop GPU the gain is across the board — roughly 11× prefill and 4× decode against the same file on CPU.

The hybrid short-conv blocks are what make a 1.2B decode at ~100 tok/s on CPU — faster than same-size pure-attention models. Long prompts prefill through the large (up to 1024-token) signatures at >1500 tok/s.

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat. GPU acceleration needs the _int4_gpu file and litert-lm ≥ 0.16.0 (litert_lm_main --backend=gpu, or the SDKs); the Gallery bundles its own runtime, which may lag that version — if its GPU toggle fails there, use CPU.

Conversion notes

Converted with released litert-torch 0.9.1 (which ships the lfm2 hybrid model support) and packaged for litert-lm ≥ 0.14, with one exporter patch: the stock LFM2 short-conv block saves its conv state from the last columns of the prefill chunk, which are padding whenever the prompt does not exactly fill a prefill signature — corrupting the first generated token of nearly every reply. The patch derives the valid length from the attention mask in-graph and gathers the conv state from the last valid columns instead; with it, engine output is token-identical to an exact per-token reference loop at every prompt length. Multi-length prefill signatures (1–1024) are exported so the runtime can pick tight chunks.

Quantization targets fully-connected + embedding ops (plus convs in the int8 file, quantized at export time); post-hoc int8 of the convs with ai-edge-quantizer breaks generation — quantize convs only at export time.

Why there is a separate _int4_gpu file. The exporter patch above reads the chunk's valid length with index_select and a mask sum, which lower to GATHER_ND and INT64 ops, and GPU delegates reject both — the CPU-only files take 536 of 579 operations on the GPU and the runtime then refuses the partial split. litert-torch 0.9.2 fixed the underlying conv-state bug upstream using valid-token masking and a one-hot matmul state select, which emits neither op. _int4_gpu is that export: litert-torch 0.9.3 + litert-converter 0.3.1, the same int4 recipe and the same weights as the int4 file, plus the executor-metadata section litert-lm ≥ 0.15 needs. It delegates fully on Android OpenCL — 501/501 and 519/519 nodes, zero rejected ops — and runs on the macOS GPU backend. Quality is unchanged: run head to head against the published int4 file on identical prompts, decoding and scoring, both score 38/50 on GSM8K (CPU, n=50), and the GPU file passes an 8-question sanity gate on both backends. The CPU-only files are left exactly as they are rather than replaced, so nothing changes for anyone already using them.

License and changes

Distributed under the LFM Open License v1.0 (see LICENSE, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.