Baekpica/Inkling-Small-Multimodal-Calibration
Viewer • Updated • 1.66k • 444 • 2
Sunghoon Baek PRO
Baekpica
AI & ML interests
Everything
Recent Activity
updated a model 1 day ago
Baekpica/Inkling-2-Small-VQ-GGUF updated a collection 2 days ago
DS4-VQ-for-DGX-Spark posted an update 2 days ago
<Why I Released the Same DeepSeek-V4.1-Flash Variant Model Three Times>
DeepSeek-V4.1-Flash-1017 is a 763B-parameter model with a 510 GB checkpoint. I compressed it into a 113.6 GB GGUF for a single 128 GB DGX Spark, retaining MTP, vision, and Engram components.
But fitting the model and preserving its behavior turned out to be two different problems.
1. VQ — Making it fit
I compressed 40 routed-expert layers using 8-dimensional vector quantization with 12/13-bit indices (1.5–1.625 index bits/weight).
It fit, but minimizing weight reconstruction error didn’t necessarily preserve model outputs.
2. iVQ — Accounting for activations
Inspired by llama.cpp’s importance matrix, I used 3.65M calibration tokens to make VQ activation-aware.
Reconstruction improved, but end-to-end quality barely changed. A tiny fraction of extreme activations dominated the importance statistics, leaving typical tokens largely unaffected.
3. hVQ — Accounting for correlations
I introduced Hessian-informed reconstruction with GPTQ-style error feedback to account for correlations between input channels.
Surprisingly, plain weight error increased, while actual model fidelity improved:
* Routed-expert output error: −34.5%
* Generated-position KL: −17%
* Top-1 agreement: 82.2% → 84.4%
* Improvements across all 40 layers
These gains were also reproduced in the actual GGUF serving engine on DGX Spark.
4. Inference optimization
Further optimizations in ds4-dfm-rs improved median prefill throughput by 74% (955.5 → 1,662.2 tok/s across 2K–64K workloads).
Decode runs at 30–36 tok/s with automatic MTP.
The takeaway: Minimizing weight error, minimizing activation-weighted error, and preserving model behavior are fundamentally different optimization problems.
At extreme compression ratios, understanding that difference matters more than simply reducing bits.
https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-hVQ-GGUF
Runtime: https://github.com/Baekpica/ds4-dfm-rs Organizations
None yet