IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-GGUF

ThinkingCap-Qwen3.8-27B-abliterated-GGUF

GGUF quantizations of ThinkingCap-Qwen3.8-27B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-Qwen3.8-27B, for llama.cpp and apps built on it.

Every file is quantized directly from the BF16 GGUF, never from another quant. All quants below Q8_0 use an importance matrix computed on this model.

Files

KLD method: llama-perplexity --kl-divergence against the BF16 GGUF on WikiText-2 test, context 2048, 60 chunks. Under 0.01 is imperceptible, 0.01–0.05 is good, above 0.1 is noticeable. Refusal behaviour was measured on the bf16 model (6/100 on Heretic's evaluator), not on each GGUF.

Why these files are larger than usual for their type

This is a hybrid model. 48 of its 64 layers are Gated DeltaNet (linear attention with a recurrent state), which holds 5.6B of the 28B parameters. Three groups of tensors are pinned to higher precision than the file's nominal type:

The pins make sizes non-monotonic across type names. Measured on these files:

• —Q5_K_S (19.1 GB) is smaller than Q4_K_M (19.5 GB). Q5 types keep DeltaNet at Q5K; Q4 types pin it to Q80.
• —Q3_K_M (18.0 GB) and IQ4_XS (18.1 GB) are the same size, as are Q3KL and Q4KS (18.8 GB).
• —Below that, sizes fall slowly: IQ3M 17.3 GB, IQ2M 15.3 GB, IQ1S 12.9 GB, because about 9 GB of every sub-4-bit file is pinned at Q80 (DeltaNet 5.9 GB, embeddings + output 2.7 GB, MTP 0.45 GB).

So choose by the KLD column at your size budget, not by the type name. Types not listed here were not better than a listed file of similar size.

Usage

llama-server, with MTP speculative decoding:

llama-server -m ThinkingCap-Qwen3.8-27B-abliterated-Q6_K.gguf \
  -ngl 99 -c 32768 -fa on --jinja --reasoning-format deepseek \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
  --chat-template-kwargs '{"reasoning_effort":"xhigh"}'

With image input, add the vision projector:

llama-server -m ThinkingCap-Qwen3.8-27B-abliterated-Q6_K.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.8-27B-abliterated-f16.gguf \
  -ngl 99 -c 32768 -fa on --jinja --reasoning-format deepseek

Notes:

• —MTP support in llama.cpp is recent and flag names have changed between versions. Check llama-server --help in your build. At the time of writing, MTP could not be combined with --mmproj or with -np above 1, so run vision and speculative decoding as separate server instances.
• —reasoning_effort accepts xhigh (default, recommended), medium or low.
• —Sampling follows Qwen3.8's recommendations, which ThinkingCap uses unchanged: thinking mode temp 1.0, top_p 0.95, top_k 20, min_p 0; non-thinking mode temp 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.
• —KV cache: only the 16 full-attention layers have one, so -ctk q8_0 -ctv q8_0 saves less memory here than on a dense 27B. Leave it at the default unless you're running long contexts.
• —Other apps (LM Studio, Ollama, etc.) work if they bundle a llama.cpp recent enough to support the Qwen3.8 (qwen3_5) architecture.

How it was made

python convert_hf_to_gguf.py <bf16-dir> --outtype bf16 --outfile model-BF16.gguf
python convert_hf_to_gguf.py <bf16-dir> --mmproj --outtype f16 --outfile mmproj-f16.gguf
llama-imatrix -m model-BF16.gguf -f calibration_datav3.txt -o imatrix.gguf -ngl 99 -c 512 --chunks 200
llama-quantize --imatrix imatrix.gguf \
  --tensor-type 'blk\.64\..*=q8_0' --tensor-type 'blk\.[0-9]+\.(attn_qkv|attn_gate|ssm_).*=q8_0' \
  model-BF16.gguf model-Q4_K_M.gguf Q4_K_M

The DeltaNet pin is dropped for Q5KS and above (those types keep DeltaNet at their own precision); the embedding/output pin applies only below Q4. Calibration text: bartowski's calibration_datav3.

Limitations

Everything from the bf16 card applies: no safety filter, 6/100 residual refusals on the bf16 model, thinking mode not separately evaluated. You are the safety layer. Lower quants add the usual degradation on long multi-step reasoning and code. If a task works on Q80 and fails on IQ4XS, the cause is the quantization.

License

PolyForm Small Business License 1.0.0 + BottleCap personal-use grant, inherited from ThinkingCap (see LICENSE). Upstream Qwen materials and the abliteration adapter are Apache-2.0 (see NOTICE). Commercial use beyond the PolyForm terms: contact BottleCap AI.

Credits

bottlecapai (ThinkingCap) · MuXodious (abliteration adapter) · p-e-w/heretic · ggml-org/llama.cpp · bartowski (imatrix calibration data) · Qwen team