Ronnied1588/Qwen2.5-7B-Instruct-heretic

Qwen2.5-7B-Instruct-heretic-GGUF

GGUF quantizations of LeadFootThrottleCock/Qwen2.5-7B-Instruct-heretic, an abliterated (decensored) version of Qwen/Qwen2.5-7B-Instruct.

Abliteration was performed using Heretic v1.2.0 with a patched configuration for AMD ROCm compatibility.

Available Quantizations

Abliteration Details

• —Tool: Heretic v1.2.0
• —Method: Optimized directional ablation with TPE-based parameter search (Optuna)
• —Selected Trial: Trial 115 (conservative selection from Pareto front)
• —Refusals: 7/100 on mlabonne/harmful_behaviors evaluation set
• —KL Divergence: 0.0820 (minimal capability degradation from base model)
• —Trials Run: 200 total (60 startup + 140 guided)
• —Batch Size: 128
• —Abliterable Components: attn.o_proj (1 per layer), mlp.down_proj (1 per layer)
• —Transformer Layers: 28

Hardware & Environment

• —GPU: AMD Radeon RX 7900 XTX (24 GB VRAM)
• —CPU: AMD Ryzen 7 7800X3D
• —RAM: 64 GB DDR5
• —OS: Linux Mint (Cinnamon)
• —PyTorch: 2.5.1+rocm6.2
• —ROCm: 6.2
• —GGUF Conversion: llama.cpp (build 8368, commit 9e2e2198b)

ROCm Compatibility Notes

Running Heretic on AMD RDNA3 GPUs requires two patches to heretic/model.py to produce correct results:

1. 1.Dedicated pad token: Heretic's default pad_token = eos_token fallback causes batched inference to produce garbage output on ROCm. Replace with a dedicated <|pad|> token and resize embeddings.

1. 1.Eager attention: Force attn_implementation="eager" in from_pretrained() to avoid SDPA backend issues on RDNA3.

Without these patches, Heretic will report nan KL divergence and meaningless refusal counts on AMD GPUs.

Usage

llama.cpp

llama-cli -m Qwen2.5-7B-Instruct-heretic-Q5_K_M.gguf -p "You are a helpful assistant." --chat-template chatml

llama.cpp server

llama-server -m Qwen2.5-7B-Instruct-heretic-Q5_K_M.gguf -ngl 99 --chat-template chatml

LM Studio / Ollama

Load any of the GGUF files directly. The chat template (ChatML) is embedded in the GGUF metadata.

Chat Template

This model uses the standard Qwen2.5 ChatML template:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Evaluation

The abliterated model was tested interactively and demonstrated:

• —Creative writing with mature themes without refusal
• —Factual chemistry/science knowledge without hedging
• —Standard coding tasks with full capability preserved
• —Balanced discussion of controversial topics without moralizing

The low KL divergence (0.0820) indicates the model retains essentially all of the original Qwen2.5-7B-Instruct capabilities while removing refusal behavior.

Credits

• —Base Model: Qwen/Qwen2.5-7B-Instruct by Alibaba Cloud
• —Abliteration Tool: Heretic by Philipp Emanuel Weidmann
• —Abliteration & Quantization: LeadFootThrottleCock

License

This model inherits the Apache 2.0 License from the base Qwen2.5-7B-Instruct model.