Ontologer/NGMI-Qwen3-8B-3Bit-Heretic

NGMI-Qwen3-8B-3Bit-Heretic

Qwen3-8B abliterated with Heretic, quantized to 3-bit (Q2_K) — the practical sweet spot between size and quality.

Heretic Abliteration

• —Base model: Qwen/Qwen3-8B
• —Method: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best
• —Refusal rate: 18/100 (down from ~99/100 baseline, 82% reduction)
• —KL divergence: 0.112
• —Abliteration applied to full-precision weights, then quantized F16 → Q2_K

Quantization

• —Format: Q2_K (K-quant mixture: attention layers ~2-bit, FFN ~3-bit)
• —Bits per weight: 3.20 bpw
• —Size: 3.1 GB (from 16.4 GB F16)
• —Speed: ~58 tok/s on NVIDIA GB10 (Blackwell)
• —Coherence: Perfect — no artifacts

Variants

This 3-bit variant is the recommended default — excellent balance of speed, size, and quality.

Usage

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Ontologer/NGMI-Qwen3-8B-3Bit-Heretic",
    filename="qwen3-8b-heretic-q2_k.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
    verbose=False
)

Acknowledgments

• —Qwen/Qwen3-8B by Alibaba
• —Heretic by p-e-w
• —llama.cpp by GGML