cognitivers/GLM-4.7-Flash-abliterated-12GB-GGUF

GLM-4.7-Flash-abliterated, 12 GB GGUF

GGUF quantizations of `huihui-ai/Huihui-GLM-4.7-Flash-abliterated`, built for one specific target: a 12 GB consumer GPU with the routed experts offloaded to system RAM.

These are not another uniform ladder. The bit budget is allocated by the role each tensor plays at inference time, which is where a Mixture-of-Experts model leaves a lot on the table.

Why a split-aware mix

GLM-4.7-Flash is a 29.94B MoE with roughly 3.3B active parameters per token. llama.cpp's --cpu-moe / --n-cpu-moe moves exactly \.ffn_(up|down|gate|gate_up)_(ch|)exps to the CPU. Everything else stays resident on the GPU forever:

The GPU-resident part is only 2.15B parameters. Keeping it at q8_0 costs about 2.3 GB, which is cheap on a 12 GB card. A uniform ladder spends the same bits per weight on those 2.15B as on the 28.2B of routed experts, and that is quality given away for nothing.

So: `q8_0` for the router, the MLA attention, the shared experts, the dense layer and the output head; `q6_k` for the token embeddings; and the routed experts compressed hard. The first four MoE layers get one extra step, since early layers tolerate low-bit worse.

Files

Both were quantized from a BF16 conversion of the original safetensors (never re-quantized from a smaller GGUF), with an importance matrix computed on calibration_datav3 over 125 chunks.

How to run it on 12 GB

# Recommended: 26.3 tok/s, 10.19 GB VRAM, 1.68 GB headroom
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -ncmoe 24 -c 8192 -fa on

# Aggressive: 33.5 tok/s, but only 0.39 GB of VRAM left. Long prompts may OOM.
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -ncmoe 20 -c 8192 -fa on

# Minimum VRAM: all experts in RAM, 2.82 GB on the card, 16.4 tok/s
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -cmoe -c 8192 -fa on

-ncmoe N keeps the routed experts of the first N layers (out of 46) in system RAM. Lower N means more experts on the GPU and more speed, until you run out of VRAM.

[image]

Measured on an RTX 4070 Ti (11874 MiB usable), context 8192, llama-bench with -p 512 -n 128 -r 2. -ncmoe 18 and below abort with a CUDA allocation error on this card.

Quality

Everything below is measured against a BF16 conversion of the same checkpoint, not against a smaller quantization, on wikitext-2-raw test (sha256 173c87a5..., 565 chunks, ctx 512).

[image]

A PPL ratio near or below 1.0 is not evidence of being better than the reference. Low-bit quantization can smooth a corpus and lower perplexity while still diverging from the original distribution, which is exactly why top-token agreement and KLD are reported next to it.

Reasoning is not free below ~3 bits per expert

GSM8K, 300 problems, paired against the same baseline, bootstrap CI over paired per-item differences:

Chain-of-thought length did not inflate (+1.8 %, upper bound +8.8 %) and tool-calling was unchanged (call rate, JSON validity and argument correctness all 1.00 in both).

Be aware of what this means: the 11.3 GB file measurably costs you reasoning accuracy. It is published because it is the best option we could measure at that size, not because it is lossless. If you can spare the system RAM, use the 18.2 GB file with -ncmoe.

Head to head at the same size

The natural comparison is mradermacher/Huihui-GLM-4.7-Flash-abliterated-i1-GGUF, a well-made imatrix ladder over the same abliterated checkpoint. Its i1-IQ3_XXS (11.65 GB) is the closest neighbour in size to our 11.30 GB file, so we are competing 0.35 GB smaller.

Both were measured in the same session, against the same BF16 base logits, with the same llama.cpp build and the same corpus. Numbers published elsewhere are not comparable to these; these are.

[image]

Smaller file, 41 % lower mean KLD, 7.1 points more top-token agreement, and 43 % less damage to reasoning. That is the case for allocating bits by role instead of uniformly.

The abliteration survives quantization

This matters more than perplexity for a model published as abliterated, and it is the one claim a quantizer can easily break without noticing.

Measured with a refusal-onset detector over StrongREJECT-small (n=60), identical protocol for all three models, including a positive control:

[image]

The control matters. A quantization scoring 0 % refusals proves nothing on its own, because a broken detector also scores 0 %. Here the same detector fires at 93.3 % on the unmodified base model, so the low scores are evidence rather than an artifact.

Scope, stated honestly: this is a refusal-onset detector built on lexical markers over the first 400 characters of the reply. It measures whether the model starts refusing. It is not the StrongREJECT harmfulness grader and it is not a safety evaluation. It supports exactly one claim, that quantization did not restore refusal behaviour, and nothing beyond that.

Reproducibility notes

• —Architecture: Glm4MoeLiteForCausalLM converts to GGUF arch deepseek2 with native MLA. The MTP head is excluded (--no-mtp) and can be exported separately as a speculative draft.
• —The Q8_0 and BF16 conversions came out byte-for-byte identical when produced independently on an A40 and on an A100 in different datacenters, so the source chain is deterministic.
• —imatrix.gguf is included so the recipe can be reproduced or extended.

Known gaps

Published deliberately with these stated rather than hidden:

• —Behavioural metrics (GSM8K, tool-calling) were measured against a Q8_0 of the same checkpoint rather than the BF16. In this model that proxy tracked the BF16-referenced KLD to within 1-9 % with an identical ranking, but it is a proxy.
• —Divergence@32 is an internal proxy set (GSM8K + MBPP + fixed prompts), not the published Divergence-300@32; absolute values are not comparable to anyone else's.
• —Long-context behaviour is untested. The model supports 202k context; nothing here was measured above 8192.
• —A Q8-vs-Q8 weight-space comparison against the unmodified base, which would quantify how much the abliteration changed, has not been run. The refusal delta above establishes it functionally.

Credits

Base model by huihui-ai, built on `zai-org/GLM-4.7-Flash`. Quantized by cognitivers. mradermacher's ladder was used as the comparison baseline and is a fine choice if you want a conventional set of sizes.