Aion-1.0-Instruct-2.3B NVFP4 (GPTQ)

NVFP4 quantization of microsoft/Aion-1.0-Instruct-2.3B, made by PureTensor from the official BF16 weights. Weights and activations are both 4-bit (W4A4). The file is 1.88 GB against 4.58 GB for BF16.

This is a test release. It is not affiliated with or endorsed by Microsoft.

Update, same day: the weights-only sibling puretensor/Aion-1.0-Instruct-2.3B-NVFP4A16 is statistically indistinguishable from BF16 on the same benchmarks. Prefer it unless you specifically need 4-bit activations.

Read this first: with 4-bit activations, a 2.3B model loses measurable quality. On our harness this checkpoint is 2.1 points below BF16 on MMLU and 6.1 points below on GSM8K, and both gaps are statistically significant. The numbers are below.

What was done

Source microsoft/Aion-1.0-Instruct-2.3B, BF16, model.safetensors sha256 6c779881...6984e
Format compressed-tensors nvfp4-pack-quantized: 4-bit E2M1 values, one FP8 E4M3 scale per group of 16, one FP32 scale per tensor
Quantized every Linear layer in the 28 transformer blocks (weights, plus 4-bit input activations)
Kept in BF16 token embeddings and the tied output head, all norms
Method GPTQ (block_size 128, dampening_frac 0.01, actorder static)
Calibration 512 samples of mlabonne/open-perfectblend, up to 2,048 tokens each
Tools llm-compressor 0.15.0, compressed-tensors 0.20.1, transformers 5.21.0, torch 2.14.1

Recipe:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head"])
oneshot(
    model=model, processor=tokenizer, recipe=recipe,
    dataset="perfectblend", splits="train[:512]",
    num_calibration_samples=512, max_seq_length=2048,
    trust_remote_code_model=True,
)
model.save_pretrained(out_dir, save_compressed=True)

Measured quality

Everything below was measured by us in one harness, with the BF16 model as the reference. The saved checkpoint was reloaded from disk and matched the in-memory quantized model exactly (identical logits on 4,096 tokens).

BF16 This checkpoint Change
MMLU, accuracy (14,042 questions) 54.88 52.74 -2.14
GSM8K, exact match (1,319 problems) 61.26 55.12 -6.14
WikiText-2 perplexity (131,072 tokens) 11.39 12.28 +0.89

Agreement with BF16, token by token:

Text Tokens Mean KL divergence 99th percentile KL Same top token
WikiText-2 test 131,072 0.095 0.71 84.7%
Chat (HuggingFaceH4/ultrachat_200k, test_sft, 256 conversations) 224,534 0.077 0.58 86.7%

Paired McNemar tests on per-question correctness against BF16: MMLU 1,058 questions lost and 758 gained (p = 2e-12); GSM8K 166 lost and 85 gained (p = 4e-7).

Details:

  • MMLU and GSM8K use the default mmlu and gsm8k task configurations of lm-evaluation-harness 0.4.13, without a chat template. GSM8K is the flexible-extract score. Our BF16 MMLU (54.88) is close to the 55.15 on the base model card; our GSM8K setup differs from theirs, so compare the two columns above with each other and not with the base card.
  • KL divergence is computed on the full next-token distribution, teacher-forced, in windows of 1,024 tokens. The chat set uses the model's chat template.

Why GPTQ

We built five NVFP4 variants from the same weights and scored them the same way. Among the first five, GPTQ was best on every metric. A later run added GPTQ with 16-bit activations, which is better still and is published separately.

Variant Activations MMLU GSM8K KL, wiki KL, chat
BF16 reference 16-bit 54.88 61.26 0 0
GPTQ (this checkpoint) 4-bit 52.74 55.12 0.095 0.077
GPTQ, weights only (sibling repo) 16-bit 54.60 61.33 0.039 0.022
Round-to-nearest 4-bit 50.55 45.56 0.175 0.327
Expanded MSE scales 4-bit 50.67 47.31 0.148 0.311
Round-to-nearest, weights only 16-bit 52.65 50.34 0.098 0.281
Expanded MSE scales, weights only 16-bit 52.32 53.37 0.098 0.263

Calibration caveat: of the 512 calibration samples, 180 are math word problems (MetaMathQA and Orca-Math; MetaMathQA is built from the GSM8K and MATH training sets) and 75 come from UltraChat, the dataset family our chat evaluation set is drawn from. The GSM8K and chat columns may therefore flatter the GPTQ rows. The MMLU and WikiText columns are unrelated to the calibration data.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "puretensor/Aion-1.0-Instruct-2.3B-NVFP4"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16
).to("cuda")

msgs = [{"role": "user", "content": "Summarise the water cycle in two sentences."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**ids, max_new_tokens=128)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

Requires compressed-tensors (pip install compressed-tensors) and trust_remote_code=True, as the base model does.

Limitations

  • No native FP4 serving yet. This architecture has no vLLM implementation at the time of writing (2026-11-20). In Transformers the 4-bit arithmetic is emulated, so this checkpoint gives the quality of NVFP4 but not its speed. All numbers above were measured that way.
  • Load in BF16. A float32 load failed in our test with a dtype mismatch. BF16 on CPU works but is slow.
  • Quality loss is real at this size. See the tables. If you need the base model's quality, use BF16.
  • English-language benchmarks only; no safety, multilingual, or long-context evaluation was run on the quantized model.

License

MIT, as the base model. The original LICENSE is included. Changes from the base release: the Linear weights are quantized to NVFP4 and stored packed, activation quantization parameters are added, and config.json gains a quantization_config block.

Downloads last month
36
Safetensors
Model size
2B params
Tensor type
BF16
ยท
U8
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for puretensor/Aion-1.0-Instruct-2.3B-NVFP4

Quantized
(13)
this model