Instructions to use puretensor/Aion-1.0-Instruct-2.3B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use puretensor/Aion-1.0-Instruct-2.3B-NVFP4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="puretensor/Aion-1.0-Instruct-2.3B-NVFP4", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("puretensor/Aion-1.0-Instruct-2.3B-NVFP4", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use puretensor/Aion-1.0-Instruct-2.3B-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "puretensor/Aion-1.0-Instruct-2.3B-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "puretensor/Aion-1.0-Instruct-2.3B-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/puretensor/Aion-1.0-Instruct-2.3B-NVFP4
- SGLang
How to use puretensor/Aion-1.0-Instruct-2.3B-NVFP4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "puretensor/Aion-1.0-Instruct-2.3B-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "puretensor/Aion-1.0-Instruct-2.3B-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "puretensor/Aion-1.0-Instruct-2.3B-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "puretensor/Aion-1.0-Instruct-2.3B-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use puretensor/Aion-1.0-Instruct-2.3B-NVFP4 with Docker Model Runner:
docker model run hf.co/puretensor/Aion-1.0-Instruct-2.3B-NVFP4
Aion-1.0-Instruct-2.3B NVFP4 (GPTQ)
NVFP4 quantization of microsoft/Aion-1.0-Instruct-2.3B, made by PureTensor from the official BF16 weights. Weights and activations are both 4-bit (W4A4). The file is 1.88 GB against 4.58 GB for BF16.
This is a test release. It is not affiliated with or endorsed by Microsoft.
Update, same day: the weights-only sibling puretensor/Aion-1.0-Instruct-2.3B-NVFP4A16 is statistically indistinguishable from BF16 on the same benchmarks. Prefer it unless you specifically need 4-bit activations.
Read this first: with 4-bit activations, a 2.3B model loses measurable quality. On our harness this checkpoint is 2.1 points below BF16 on MMLU and 6.1 points below on GSM8K, and both gaps are statistically significant. The numbers are below.
What was done
| Source | microsoft/Aion-1.0-Instruct-2.3B, BF16, model.safetensors sha256 6c779881...6984e |
| Format | compressed-tensors nvfp4-pack-quantized: 4-bit E2M1 values, one FP8 E4M3 scale per group of 16, one FP32 scale per tensor |
| Quantized | every Linear layer in the 28 transformer blocks (weights, plus 4-bit input activations) |
| Kept in BF16 | token embeddings and the tied output head, all norms |
| Method | GPTQ (block_size 128, dampening_frac 0.01, actorder static) |
| Calibration | 512 samples of mlabonne/open-perfectblend, up to 2,048 tokens each |
| Tools | llm-compressor 0.15.0, compressed-tensors 0.20.1, transformers 5.21.0, torch 2.14.1 |
Recipe:
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
recipe = GPTQModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head"])
oneshot(
model=model, processor=tokenizer, recipe=recipe,
dataset="perfectblend", splits="train[:512]",
num_calibration_samples=512, max_seq_length=2048,
trust_remote_code_model=True,
)
model.save_pretrained(out_dir, save_compressed=True)
Measured quality
Everything below was measured by us in one harness, with the BF16 model as the reference. The saved checkpoint was reloaded from disk and matched the in-memory quantized model exactly (identical logits on 4,096 tokens).
| BF16 | This checkpoint | Change | |
|---|---|---|---|
| MMLU, accuracy (14,042 questions) | 54.88 | 52.74 | -2.14 |
| GSM8K, exact match (1,319 problems) | 61.26 | 55.12 | -6.14 |
| WikiText-2 perplexity (131,072 tokens) | 11.39 | 12.28 | +0.89 |
Agreement with BF16, token by token:
| Text | Tokens | Mean KL divergence | 99th percentile KL | Same top token |
|---|---|---|---|---|
| WikiText-2 test | 131,072 | 0.095 | 0.71 | 84.7% |
Chat (HuggingFaceH4/ultrachat_200k, test_sft, 256 conversations) |
224,534 | 0.077 | 0.58 | 86.7% |
Paired McNemar tests on per-question correctness against BF16: MMLU 1,058 questions lost and 758 gained (p = 2e-12); GSM8K 166 lost and 85 gained (p = 4e-7).
Details:
- MMLU and GSM8K use the default
mmluandgsm8ktask configurations of lm-evaluation-harness 0.4.13, without a chat template. GSM8K is the flexible-extract score. Our BF16 MMLU (54.88) is close to the 55.15 on the base model card; our GSM8K setup differs from theirs, so compare the two columns above with each other and not with the base card. - KL divergence is computed on the full next-token distribution, teacher-forced, in windows of 1,024 tokens. The chat set uses the model's chat template.
Why GPTQ
We built five NVFP4 variants from the same weights and scored them the same way. Among the first five, GPTQ was best on every metric. A later run added GPTQ with 16-bit activations, which is better still and is published separately.
| Variant | Activations | MMLU | GSM8K | KL, wiki | KL, chat |
|---|---|---|---|---|---|
| BF16 reference | 16-bit | 54.88 | 61.26 | 0 | 0 |
| GPTQ (this checkpoint) | 4-bit | 52.74 | 55.12 | 0.095 | 0.077 |
| GPTQ, weights only (sibling repo) | 16-bit | 54.60 | 61.33 | 0.039 | 0.022 |
| Round-to-nearest | 4-bit | 50.55 | 45.56 | 0.175 | 0.327 |
| Expanded MSE scales | 4-bit | 50.67 | 47.31 | 0.148 | 0.311 |
| Round-to-nearest, weights only | 16-bit | 52.65 | 50.34 | 0.098 | 0.281 |
| Expanded MSE scales, weights only | 16-bit | 52.32 | 53.37 | 0.098 | 0.263 |
Calibration caveat: of the 512 calibration samples, 180 are math word problems (MetaMathQA and Orca-Math; MetaMathQA is built from the GSM8K and MATH training sets) and 75 come from UltraChat, the dataset family our chat evaluation set is drawn from. The GSM8K and chat columns may therefore flatter the GPTQ rows. The MMLU and WikiText columns are unrelated to the calibration data.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "puretensor/Aion-1.0-Instruct-2.3B-NVFP4"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, trust_remote_code=True, dtype=torch.bfloat16
).to("cuda")
msgs = [{"role": "user", "content": "Summarise the water cycle in two sentences."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**ids, max_new_tokens=128)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
Requires compressed-tensors (pip install compressed-tensors) and trust_remote_code=True, as the base model does.
Limitations
- No native FP4 serving yet. This architecture has no vLLM implementation at the time of writing (2026-11-20). In Transformers the 4-bit arithmetic is emulated, so this checkpoint gives the quality of NVFP4 but not its speed. All numbers above were measured that way.
- Load in BF16. A float32 load failed in our test with a dtype mismatch. BF16 on CPU works but is slow.
- Quality loss is real at this size. See the tables. If you need the base model's quality, use BF16.
- English-language benchmarks only; no safety, multilingual, or long-context evaluation was run on the quantized model.
License
MIT, as the base model. The original LICENSE is included. Changes from the base release: the Linear weights are quantized to NVFP4 and stored packed, activation quantization parameters are added, and config.json gains a quantization_config block.
- Downloads last month
- 36
Model tree for puretensor/Aion-1.0-Instruct-2.3B-NVFP4
Base model
microsoft/Aion-1.0-Instruct-2.3B