YuE2-3B-FP8 (autoregressive checkpoint, W8A8)

This is an FP8 (W8A8, dynamic per-token activations) requantisation of the autoregressive (AR) half of m-a-p/YuE2-3B. It is packaged as the Qwen3-shaped checkpoint that the official yue2_infer pipeline derives internally for its vLLM backend, and it is a drop-in replacement for that derived checkpoint. The planning (abc) and semantic-token stages run on these weights. The NAR flow-matching stage and the VAE still run on the original bf16 and fp32 weights, which you need alongside this repo.

The point of it is decode speed on bandwidth-limited hardware. On a DGX Spark (GB10, 273 GB/s unified memory) the AR stages are limited by weight bandwidth, and the bf16 AR checkpoint is 4.33 GB.

	bf16 (derived by yue2_infer)	this checkpoint
AR weights on disk	4.33 GB	2.92 GB
Linear layers	bf16	FP8 e4m3 weights with a scale per output channel, FP8 e4m3 activations scaled per token at run time
embed_tokens and lm_head (vocab 184,704)	bf16	bf16, not quantised
AR decode on DGX Spark, vLLM 0.27.1, batch 1	44.8 tok/s abc, 43.0 tok/s semantic	79 to 81 tok/s abc, 67 to 76 tok/s semantic (1.8x)

This build is the fidelity option. The NVFP4 sibling (DKmode22/YuE2-3B-NVFP4) is smaller and faster on unified-memory hardware at a somewhat larger deviation from bf16; the numbers for both are below.

What was quantised and how

Source: model.safetensors from m-a-p/YuE2-3B (sha256 starting 1d55c42c). The AR tensors were extracted exactly as yue2.fast.derive_ar_checkpoint does: 311 tensors covering attention q/k/v/o and their q/k norms, the AR MLPs, layer norms, embed_tokens and lm_head. The NAR MLPs, the llm2vae and vae2llm projections, the timestep embedder and the VAE are not in this checkpoint and are not quantised.

Tool: llm-compressor 0.13.0 with compressed-tensors 0.18.0. Recipe:

QuantizationModifier(targets=["Linear"], scheme="FP8_DYNAMIC", ignore=["lm_head"])

Calibration used sequences the bf16 model generated itself, not a text corpus, because text never exercises the codec-token activations. The set was 32 requests (16 lyric sets, each rendered under two contrasting style prompts, English and Mandarin, cot=full and cot=melody), both the abc and semantic phases, cut into 58 windows of up to 8,192 tokens: 200K tokens in total, 51% of them codec tokens. Four requests were held out for evaluation.

Format: compressed-tensors float-quantized. Loads in vLLM on any GPU with FP8 support (Ada, Hopper, Blackwell including DGX Spark), and in transformers with the weights dequantised to bf16.

Quality against bf16

Measured through vLLM 0.27.1 on the four held-out requests, with every output position teacher-forced on the bf16 model's own sampled sequences: 6,286 abc positions and 17,931 semantic positions. The first row is two bf16 runs on different engine builds, which gives the numerical floor.

Top-1 agreement is a poor guide for the semantic phase. The codec distribution is nearly flat at most positions (bf16 itself puts its argmax on the token it sampled only 22% of the time), so a small shift flips the argmax without moving much probability. The last column, total-variation distance, is the share of sampling probability that actually moved, and is the number to read.

Checkpoint	abc top-1 agree	abc NLL	semantic top-1 agree	semantic NLL	semantic TV
bf16 vs bf16, two engine builds	0.999	0.141 / 0.141	0.982	3.414 / 3.414	1.2%
this checkpoint, FP8 W8A8	0.997	0.141 / 0.141	0.942	3.414 / 3.418 (+0.1%)	3.8%
NVFP4 W4A16 GPTQ, sibling repo DKmode22/YuE2-3B-NVFP4	0.996	0.141 / 0.142	0.905	3.414 / 3.424 (+0.3%)	6.3%
NVFP4 W4A16, AWQ	0.993	0.141 / 0.143	0.896	3.414 / 3.433 (+0.6%)	7.4%
NVFP4 W4A4, GPTQ (not published)	0.992	0.141 / 0.144	0.854	3.414 / 3.446 (+0.9%)	10.1%

Split by how confident bf16 was at each position, both published checkpoints agree with bf16 at more than 99.8% of positions where the bf16 top-1 probability is above 0.5; the disagreements sit in near-tie positions. Streams were checked for repetition: no repeated 24-gram on any held-out output.

How to use

The official pipeline derives its own bf16 AR checkpoint and does not yet accept an alternative. patch_fast.py in this repo adds a YUE2_AR_CHECKPOINT environment override to yue2/fast.py. It is idempotent and changes nothing else.

pip install "yue2_infer[fast]"        # the wheel from m-a-p/YuE2-3B
python patch_fast.py "$(python -c 'import yue2.fast,os;print(os.path.abspath(yue2.fast.__file__))')"
hf download DKmode22/YuE2-3B-FP8 --local-dir ./yue2-3b-fp8
export YUE2_AR_CHECKPOINT=$PWD/yue2-3b-fp8
python - <<'EOF'
from yue2 import YuE2Pipeline
pipe = YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", backend="vllm", memory_budget_gib=16)
song = pipe(style="Indie folk, warm acoustic guitar, male vocal", lyrics="[Verse]\n...")
song.save("song.flac")
EOF

Notes:

• The pipeline uses vLLM only when cfg_scale is 1 (the default for cot=full and cot=melody) and quantization is "none". With cot=off the guidance is 1.01 and the pipeline silently falls back to its torch backend, which does not use this checkpoint.
• Measured on DGX Spark (sm_121) with vLLM 0.27.1. The package pins vllm==0.19.0; to use a newer build, widen the version check in yue2/fast.py. The 0.19 wheel's FP8 path was not tested on sm_121.
• Everything else in the pipeline is unchanged: one sequence at a time, the windowed repetition penalty, 24,576 context.

Licence and attribution

The base model is released by the M-A-P team under CC-BY-NC-4.0. This derivative carries the same licence and is for non-commercial use. Credit for the model belongs to the YuE2 authors; please cite their paper (arXiv 2503.08638). Quantisation, calibration corpus and evaluation by DKmode22 on DGX Spark hardware, September 2026.