Modelpublic

sunnyyy/qwen38-27b-huihui-slim-lora

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

1likes30downloads

Model Card

Qwen3.8-27B × Huihui Abliterated — Slim LoRA (r=64)

The delta between huihui-ai/Huihui-Qwen3.8-27B-abliterated and Qwen/Qwen3.8-27B, extracted as a rank-64 LoRA and energy-pruned from 6.1 GB → 203 MB (bf16) / 108 MB (q8_0 GGUF).

Only the 98 modules whose weights actually changed are kept: linear_attn.out_proj × 36, mlp.down_proj × 49, self_attn.o_proj × 13. Zero-diff modules, the vision tower and full-rank embed/lm_head weights were pruned away.

Files

path	what it is	size
`lora/`	PEFT adapter (transformers / vLLM), bf16, r=64, alpha=64	203 MB
`gguf/huihui-slim-lora-q8_0.gguf`	LoRA-GGUF for llama.cpp runtime mounting (`--lora`)	108 MB
`gguf/huihui-slim-lora-bf16.gguf`	lossless bf16 LoRA-GGUF (verification baseline)	203 MB

Quickstart — llama.cpp (runtime LoRA mounting, not baked in)

The base must be a Qwen3.8-27B GGUF (general.architecture = qwen35), e.g. ggml-org/Qwen3.8-27B-GGUF or any of your own quants:

bash

llama-server -m Qwen3.8-27B-Q8_0.gguf \
             --lora-scaled gguf/huihui-slim-lora-q8_0.gguf:0.1 \
             -ngl 99 -c 16384 --jinja

Scale = how strongly the abliteration delta is applied:

scale	effect
`0.1` (recommended starting point)	light touch, behavior stays close to the base
`0.5`	moderate
`1.0` (plain `--lora FILE`)	full-strength abliteration

Multiple adapters can be stacked: --lora-scaled a.gguf:0.5,b.gguf:1.0.

⚠️ If you get garbage output ((begin-#: (begin, ...), first verify the base file is really Qwen3.8-27B: llama-gguf info <file> | grep -E "architecture|name".

Quickstart — transformers / vLLM (PEFT)

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B", torch_dtype="auto")
model = PeftModel.from_pretrained(base, "sunnyyy/qwen38-27b-huihui-slim-lora/lora")  # or local path

Verification record

—bf16 GGUF is bit-exact vs the PEFT adapter (incl. the Qwen3.5 V-head column permutation on out_proj)
—q80 GGUF dequantized and compared tensor-by-tensor against the bf16 build: max relative error 4.1e-3 (normal Q80 noise)
—runtime verified: llama.cpp + Qwen3.8-27B UD-Q6_K base @ scale 0.1, inference clean

Generation pipeline (fully reproducible)

Everything used to produce this adapter — patched mergekit (Qwen3.5 dense support + tmpfs streaming extraction), energy-pruning slimmer, and the LoRA→GGUF converter with the Qwen3.5 V-head reorder fix:

https://github.com/sunnyyangyangyang/qwen38-27b-abliterated-lora