CoolFace
Modelpublic

keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-GGUF

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
0likes127downloads
Model Card

Qwen3.6-35B-A3B-REAM-192-heretic — GGUF

⚠️ Preliminary release. Quantized from the safetensors output of the Heretic optimization run; the only validation performed so far is (a) Heretic's own first-token KL + refusal-marker eval against its 100-prompt evaluation split, and (b) a five-prompt qualitative chat spot-check. No academic benchmarks (MMLU, GSM8K, IFEval, HumanEval), agentic-harness benchmarks, or formal capability comparisons against the source [REAM-192](https://huggingface.co/keithnull/Qwen3.6-35B-A3B-REAM-192) base have been run yet. Treat this release as a preview while those evaluations are pending. The model card will be updated with hard numbers as they land.

GGUF quantizations of keithnull/Qwen3.6-35B-A3B-REAM-192-heretic, the Heretic-abliterated variant of keithnull/Qwen3.6-35B-A3B-REAM-192. 27.05B parameters, 192 routed experts (REAM-merged from 256), with the refusal direction ablated via Heretic v1.3.0 + a custom fork that adds full Qwen3.5/3.6 MoE coverage.

For APEX MoE-aware mixed-precision quants of the same model, see keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-APEX-GGUF.

What has been validated so far

CheckResultSource
Heretic eval-set refusals (out of 100 harmful prompts)✅ 10 / 100 (vs ~80 / 100 baseline)Heretic optimization run, trial 67
First-token KL divergence (harmless eval prompts)0.0008Heretic optimization run, trial 67
Qualitative chat spot-checkPassed on the hardest of 5 refusal-trigger prompts (Instagram cookie-hijacking scrape) — substantive technical answer with intact thinking-mode reasoning, no soft-refusal lecture-stackingManual probe of the saved bf16 model

That is the entire validation set for this preliminary release. The Heretic refusal/KL numbers are computed against the small in-harness evaluation prompt set bundled with Heretic; they confirm the abliteration mechanically did its job and that capability damage at the first-token level is minimal, but they are not equivalent to running standard academic benchmarks against the deployed quant.

Available quantizations

FileSizeBPWNotes
Qwen3.6-35B-A3B-REAM-192-heretic-Q4_K_M.gguf16 GB4.85Recommended daily driver. Good balance of quality and footprint; fits comfortably on 24 GB VRAM with usable context.
Qwen3.6-35B-A3B-REAM-192-heretic-Q3_K_S.gguf11 GB3.52Smaller fallback. Fits on 16 GB VRAM. Some quality loss vs Q4KM but still very usable.
Qwen3.6-35B-A3B-REAM-192-heretic-bf16.gguf51 GB16.01Full-precision GGUF, intermediate. Useful as a master from which to derive new quant levels (Q5KM, IQ3M, IQ4XS, etc) without re-running convert_hf_to_gguf.py.
mmproj-REAM-192-heretic-F16.gguf~2-3 GB—Vision tower sidecar for llama-mtmd-cli / llama-server multimodal mode. Vision capabilities preserved from the source Qwen3.6-VL base — Heretic only touches the language model.

Quantization details

Heretic-specific configuration (see source repo for full provenance):

Trial 67 abliteration parameters

ParameterValue
direction_index17.12
attn.o_proj.max_weight1.49
attn.o_proj.max_weight_position29.14
attn.o_proj.min_weight0.77
attn.o_proj.min_weight_distance5.21
attn.out_proj.max_weight0.95
attn.out_proj.max_weight_position26.92
attn.out_proj.min_weight0.78
attn.out_proj.min_weight_distance22.12
mlp.down_proj.max_weight1.50
mlp.down_proj.max_weight_position26.81
mlp.down_proj.min_weight1.11
mlp.down_proj.min_weight_distance22.33

Quick start

Text-only inference

bash
./llama-cli \
  -m Qwen3.6-35B-A3B-REAM-192-heretic-Q4_K_M.gguf \
  -p "Write a Python function that returns the nth Fibonacci number." \
  -n 200 --temp 0.0 --n-gpu-layers 999