Modelpublic

arxorry/GLM-5.1-Abliterated-Q5_K_M-GGUF

sourceHugging Faceagpl-3.0updated 5mo agoView on Hugging Face

0likes43downloads

Model Card

GLM-5.1-Abliterated Q5KM GGUF

TL;DR

—For agentic / long-horizon / coding workloads where you don't want refusals on technical questions
—754B MoE (40B active), abliterated, Q5_K_M, 534 GB / 33 shards
—~37 t/s decode for ~$8-10/hr on 8× RTX PRO 6000 Blackwell (vast.ai)
—~23 t/s decode for ~$8-10/hr on 8× A100-SXM4 (vast.ai)
—Full 200K context tested, with q8_0 KV cache
—AIME 2026: 6/6 correct on the first six problems (I did partial run with thinking-on)
—8× 80GB or 8× 96GB GPU for the recommended config

GGUF Q5KM quantization of `helixdouble/GLM-5.1-Abliterated`. Runs in mainline llama.cpp, LM Studio, Ollama.

Abliterated means the model was modified to reduce refusal behavior. It removes the directions in the model's activation space that produce refusals, without retraining. The result is a model that declines less often on prompts it would otherwise refuse, while keeping general capability intact. This quant inherits that behavior from upstream — no additional abliteration was applied here.

The model will engage with technical questions that mainstream chat models often over-refuse (security research, defensive tooling, dual-use). However, this model still declines to provide methods on self-harm queries and similar.

Performance

--parallel 1, q8_0 KV cache, flash attention on.

8× RTX PRO 6000 Blackwell 96GB

ctx=202752

Test	Prefill t/s	Decode t/s
256-token gen, short prompt	215	36.80
8k prompt + 128 gen	659	34.05
32k prompt + 128 gen	502	28.57
128k prompt + 128 gen	240	17.91
short prompt + 2k gen	295	36.14

~$8-10/hr on vast.ai. huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated or similar model also fits in VRAM simultaneously - you can use it for subagent tasks!

8× A100-SXM4 80GB

ctx=202752

Test	Prefill t/s	Decode t/s
short prompt, 256 gen	55	23.27
short prompt, 512 gen	59	23.00
short prompt, 1024 gen	57	22.83
4.5k prompt + 128 gen	209	22.43

~$8-10/hr on vast.ai.

2× RTX PRO 6000 Blackwell 96GB

ctx=8192, mostly offloaded to RAM

Run	Output tokens	Prefill t/s	Decode t/s
prompt 60 tokens, 256 gen	256	18.56	5.43
prompt 67 tokens, 512 gen	512	19.65	5.33
prompt 2222 tokens, 128 gen	66	81.48	5.37
prompt 45 tokens, no max_tokens	1976	14.98	5.47

~$3-5/hr on vast.ai.

Full-config numbers above are the recommended hardware target. Two-GPU operation works but the model does not fully fit in VRAM at this configuration.

Quality — AIME 2026

Just to quickly check nothing went wrong, I ran partial evaluation of AIME 2026.

1 attempt per problem, sampling temperature=1.0 top_p=0.95, enable_thinking=true, default matharena evaluator.

Problem	Wall time	Reasoning tokens	Result
1	0m 48s	1,089	✓
2	6m 08s	8,117	✓
3	5m 16s	6,987	✓
4	9m 05s	11,869	✓
5	2m 19s	3,131	✓
6	1m 29s	2,005	✓

6/6 correct on the first 6 problems consecutively.

Quick start

1.Start llama-server and let it download the GGUF from Hugging Face:

bash

llama-server \
  --hf-repo arxorry/GLM-5.1-Abliterated-Q5_K_M-GGUF \
  --hf-file GLM-5.1-Abliterated-Q5_K_M-00001-of-00033.gguf \
  --ctx-size 202752 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

1.Open the llama.cpp web UI or connect with an OpenAI-compatible client.

1.For manual downloads, keep all 33 shards in the same directory and load the first shard.

Prompt Format

GLM-5.1 uses its own chat template with explicit <think> / </think> blocks for the reasoning trace. The full jinja template is embedded in the GGUF.

Token markers:

[gMASK]<sop><|system|>system prompt<|user|>user message<|assistant|><think>reasoning</think>visible answer

Thinking mode is on by default. To disable, pass enable_thinking=false in chat_template_kwargs:

json

{
  "messages": [{"role": "user", "content": "Hello"}],
  "chat_template_kwargs": {"enable_thinking": false}
}

OpenAI-compatible clients work directly with the /v1/chat/completions endpoint.

Source

This was quantized from `helixdouble/GLM-5.1-Abliterated`, which is based on `zai-org/GLM-5.1-FP8` / `zai-org/GLM-5.1`. This release changes the storage/runtime format and quantization only.

Quantization Recipe

Direct Q5KM from BF16 via llama-quantize. No imatrix calibration, no per-tensor overrides. Reproducible end-to-end from helixdouble/GLM-5.1-Abliterated FP8 source with a single command.

Conversion timings (CPU-only, 128 threads, ~3 TB scratch disk):

—HF safetensors -> BF16 GGUF: 55m 14s
—BF16 GGUF -> Q5KM GGUF: 31m 37s

Notes

—Benchmarks above are single-client. I haven't run concurrent-client throughput.
—For production deployment add your own guardrails - abliterated != aligned.

Feedback

Open an issue in the Community tab. This is my first quant, feedback is genuinely useful😀

Support

If this quant saved you some vast.ai bucks, a tip helps fund the next one.

Disclaimer

Provided AS IS for research and educational purposes. This model has reduced refusal behavior inherited from upstream - outputs may be inaccurate, biased, unsafe, or that you find may offensive. You are responsible for compliance with applicable laws in your jurisdiction and for any guardrails you add when deploying.

No warranty is given, no liability accepted for downstream use.

License

This GGUF follows the source model licensing. The source model is listed as AGPL-3.0 and also refers users to the upstream GLM-5.1-FP8 license. For redistribution, modification, or hosted use, check the upstream model cards and license files.