Gemma4-E4B-it-MMFlash

Paper · Project page · Code

MMFlash draft model for speculative decoding of google/gemma-4-E4B-it on image and text inputs. It is not a standalone language model; it is paired with the target model in a speculative-decoding server (SGLang DFLASH).

MMFlash builds on DFlash block-diffusion drafting: the draft reads hidden states from five layers of the target and proposes a block of 16 tokens in one forward pass, and the target verifies the block in one forward pass. MMFlash keeps the DFlash architecture and is trained on multimodal data with a loss weighted by per-token visual-grounding (visual-KL) scores.

Qwen3.5-9B on a MathVista question, recorded in real time on one NVIDIA H200 with SGLang (greedy decoding).

Need a drafter for another model? Email us at fscnkucs@gmail.com and we can help train one. We will keep updating these releases, and a Qwen4 checkpoint is planned.

Model details

Target model google/gemma-4-E4B-it
Parameters 537M (bf16)
Block size 16
Target hidden states layers 1, 10, 20, 30, 39 (of 42)

Usage

SGLang

SGLANG_GEMMA_OUT_OF_PLACE_POSITION_MUTATION=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
python -m sglang.launch_server \
  --model-path google/gemma-4-E4B-it \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path FSCCS/Gemma4-E4B-it-MMFlash \
  --speculative-dflash-block-size 16 \
  --attention-backend triton \
  --mm-attention-backend sdpa \
  --mem-fraction-static 0.7 \
  --reasoning-parser gemma4 \
  --trust-remote-code

vLLM

vllm serve google/gemma-4-E4B-it \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --reasoning-parser gemma4 \
  --speculative-config '{"method": "dflash", "model": "FSCCS/Gemma4-E4B-it-MMFlash", "num_speculative_tokens": 15}'

Tested with vLLM 0.32.0, which serves DFlash drafts natively. num_speculative_tokens is the block size minus one. vLLM does not have the gemma4-spec changes, so τ is lower than in the results below (ChartQA: 5.71 vs 6.39).

Results

Each cell is τ / speedup: τ is the average acceptance length (tokens accepted per verification step), and the speedup is the decoding throughput over Gemma 4 E4B-it without speculative decoding.

Temperature 0

Method ChartQA MMStar DynaMath
MTP (5, official) 3.81 / 1.78× 3.19 / 1.48× 4.04 / 1.89×
MTP (15) 4.73 / 1.66× 3.74 / 1.33× 5.17 / 1.75×
EAGLE-3 (16) 4.87 / 2.27× 4.04 / 1.50× 6.23 / 1.90×
MMFlash (16) 6.39 / 2.87× 4.83 / 2.10× 7.15 / 3.02×

Values in parentheses are the number of draft steps for MTP, the draft tree size for EAGLE-3, and the block size for MMFlash. MTP (5, official) is Google's released drafter, google/gemma-4-E4B-it-assistant, run for 5 steps; MTP (15) runs it for 15 steps.

Concurrency (temperature 0)

MMFlash at 1, 8 and 32 requests in flight, on a single NVIDIA H200. Each cell is the decode throughput in tokens/s, with the speedup over Gemma 4 E4B-it alone at the same concurrency in parentheses.

Concurrency ChartQA CharXiv MMStar MMBench DynaMath MathVista MathVerse Avg.
1 521 (2.87×) 448 (2.50×) 377 (2.10×) 363 (1.99×) 543 (3.02×) 454 (2.53×) 436 (2.45×) 449 (2.49×)
8 2,819 (2.42×) 2,491 (2.16×) 2,129 (1.89×) 1,986 (1.75×) 3,068 (2.60×) 2,391 (2.25×) 2,484 (2.12×) 2,481 (2.17×)
32 5,230 (1.68×) 4,957 (1.64×) 4,011 (1.49×) 3,948 (1.25×) 5,199 (1.52×) 4,332 (1.52×) 5,122 (1.46×) 4,686 (1.51×)

Citation

@article{feng2026mmflash,
  title   = {MMFlash: Faster Speculative Decoding for Vision-Language Models},
  author  = {Feng, Sicheng and Chen, Zigeng and Wang, Song and Fang, Gongfan and Wang, Xinchao},
  journal = {arXiv preprint arXiv:2611.18482},
  year    = {2026}
}
Downloads last month
65
Safetensors
Model size
0.5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for FSCCS/Gemma4-E4B-it-MMFlash

Finetuned
(484)
this model

Collection including FSCCS/Gemma4-E4B-it-MMFlash

Paper for FSCCS/Gemma4-E4B-it-MMFlash