steampunque/GLM-4.7-Flash-MP-GGUF

Mixed Precision GGUF layer quantization of GLM-4.7-Flash by zai-org

Original model: https://huggingface.co/zai-org/GLM-4.7-Flash

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file an extended layer definition E quant Q4_E_H is defined as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_666"],[1 ,"Q5_K_555"],[2 ,"Q4_K_555"],[3 ,"Q4_K_555"],[4 ,"Q4_K_555"],[5 ,"Q4_K_554"],[6 ,"Q4_K_554"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_554"],[9 ,"Q4_K_554"],[10,"Q4_K_554"],[11,"Q4_K_554"],[12,"Q4_K_554"],[13,"Q4_K_554"],[14,"Q4_K_554"],[15,"Q4_K_554"],
   [16,"Q4_K_555"],[17,"Q4_K_555"],[18,"Q4_K_555"],[19,"Q4_K_555"],[20,"Q4_K_555"],[21,"Q4_K_555"],[22,"Q4_K_555"],[23,"Q4_K_555"],
   [24,"Q4_K_555"],[25,"Q4_K_555"],[26,"Q4_K_555"],[27,"Q4_K_555"],[28,"Q4_K_555"],[29,"Q4_K_555"],[30,"Q4_K_555"],[31,"Q4_K_555"],
   [32,"Q4_K_555"],[33,"Q4_K_555"],[34,"Q4_K_555"],[35,"Q4_K_555"],[36,"Q4_K_555"],[37,"Q4_K_555"],[38,"Q4_K_555"],[39,"Q4_K_555"],
   [40,"Q4_K_655"],[41,"Q4_K_655"],[42,"Q4_K_655"],[43,"Q5_K_655"],[44,"Q5_K_666"],[45,"Q5_K_668"],[46,"Q6_K_866"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The quant was evaluated for strong reasoning performance across a curated set of test prompts. Experimenting with different quant levels determined this model to be quite sensitive to losing coherent reasoning ability when quantization to lower bit precision levels. The Q4_E_H quant optimization outperformed previous Q4_K_H and Q6_K_H quants noticeably on the test prompts so they have been deleted. The Q4_E_H quant is sized to be able to run in 24G VRAM with some space left for context. In tests the model still shows significant overthinking on some problems. It fared well against some simple IQ test type problems evidencing a fairly strong latent space / well curated traing set for only 30G class model. If a problem does not fall into its latent solution space it can just fall into a endless rep loop typical of other thinking models. It clearly uses forced introspections in its RL training as an attempt to increase its gen reliability and if the model is unable to find consistency in the forced double/triple/quadraple+ checks it can get stuck in reasoning loops with greedy sampling.

The original Q4_K_H quant made for GLM-4.7-Flash is restored here due to anomalous performance of Q4_E_H on evals. Due to fragility of this model vs. quantization it appears extremely difficult to generate a universally good quant; i.e. Q4_E_H does great on optimization prompt set but shows worse performance on evals vs. Q4_K_H does not do so well on optimization prompt set but does noticeably better on some evals. So Q4_K_H will be restored. It is defined using E quant MP layer definition syntax as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_555"], [1 ,"Q4_K_466"], [2 ,"Q4_K_446"], [3 ,"Q4_K_445"], [4 ,"Q4_K_445"], [5 ,"Q4_K_444"], [6 ,"Q4_K_444"], [7 ,"Q4_K_444"],
   [8 ,"Q4_K_444"], [9 ,"Q4_K_444"], [10,"Q4_K_444"], [11,"Q4_K_444"], [12,"Q4_K_444"], [13,"Q4_K_444"], [14,"Q4_K_444"], [15,"Q4_K_444"],
   [16,"Q4_K_444"], [17,"Q4_K_444"], [18,"Q4_K_444"], [19,"Q4_K_444"], [20,"Q4_K_444"], [21,"Q4_K_444"], [22,"Q4_K_444"], [23,"Q4_K_444"],
   [24,"Q4_K_444"], [25,"Q4_K_444"], [26,"Q4_K_444"], [27,"Q4_K_444"], [28,"Q4_K_446"], [29,"Q4_K_444"], [30,"Q4_K_444"], [31,"Q4_K_444"],
   [32,"Q4_K_444"], [33,"Q4_K_444"], [34,"Q4_K_446"], [35,"Q4_K_444"], [36,"Q4_K_444"], [37,"Q4_K_446"], [38,"Q4_K_444"], [39,"Q4_K_444"],
   [40,"Q4_K_446"], [41,"Q4_K_446"], [42,"Q4_K_466"], [43,"Q5_K_555"], [44,"Q5_K_556"], [45,"Q5_K_566"], [46,"Q6_K_666"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

Comparison:

Quant	size	PPL	Comment
Q4_K_M	18.1e9	11.0	Q4 embed Q6 out
Q4_E_H	18.5e9	10.7	Hybrid E quant with Q6 embed Q6 out
Q4_K_H	18.0e9	11.0	Hybrid MP K quant with Q6 embed Q6 out

Usage:

This is a RL trained Moe thinking model. The layer quants for this model were optimized over a set of test/eval prompts using greedy sampling. The model overthinks and occasionally get stuck in infinite generations. When it can solve a problem it normally does a very good job. It appears to have been trained with forced introspections and possibly given less reward for efficient solutions than other GLM based thinkers as it will spit out a ton of tokens with many unecessary reflections.

To bypass think mode inject a think start and stop at the beginning of gen or modify the assistant prompt template to add THINK_START and THINK_STOP at the end of the assistant template:

THINK_START="<think>"
THINK_STOP="</think>"

These tokens must be tokenized as special tokens.

The model can be run on consumer grade hardware using tensor offload to CPU i.e.

OT="-ot exps=CPU -ngl 99"

The model should not be speculated when using dominant CPU offload since CPU does not have enough parallel hardware to benefit in large speed increase from processing larger batches of tokens. Negligible speedup was found Using Qwen3 0.6 as a speculator so speculation is also not recommended with full GPU offload.

It is possible to partially offloading experts to CPU i.e. experts 18-46 only are offloaded to CPU:

OT="-ot blk\.1[8-9]|2[0-9]|3[0-9]|4[0-6].*exps=CPU -ngl 99"

This partial offload is not recommended as it results in slower gen most likely related to extra shuffling of data between GPU and CPU in this mode, the exact cause of inefficiency was not investigated since -ot exps=CPU gives very usable gen rates by itself even on an older CPU (9900k/DDR4 mem).

Approx performance with llama.cpp b9404 usings 4070 (12G VRAM) GPUs with 9900k (128G RAM) CPU :

CONFIG	QKV	NKV	gen tps	Comment
-ot exps=CPU	F16	180k	21	llama.cpp 9404
-ot exps=CPU	Q8_0	198k	21	""
-ot exps=CPU 18-46	F16	80k	18	""
RPC (2x4070)	F16	80k	76	""
" "	Q8_0	144k	75	""

Long context test from https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt was run at Q8_0 QKV and although the model was heading toward the right answer it then got stuck in a long rep loop. It handled a simple needle in haystack long context test with no issue. Prompt processing slows down to a crawl heading toward 100k+ tokens even with full GPU offload over RPC.

The model was tested against a small set of code gen prompts using greedy sampling and found to be quite strong in its ability to generate working programs with both think mode enabled or disabled suggesting it was intentionally tuned for a strong coding base in its pretrain/postrain/RL optimization. It got stuck in a rep loop during code gen on one of the test prompts but shutting off think mode allowed the model to successfully generate working code on the test prompt.

Benchmarks:

Math and code benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
GLM-4.7-Flash.Q4_E_H.gguf	Q4_E_H	18.5e9 B	~0.4B bigger than Q4_K_M
GLM-4.7-Flash.Q4_K_H.gguf	Q4_K_H	18.0e9 B	~Q4_K_M size

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040