CoolFace
Modelpublic

Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
7likes8.3kdownloads
Model Card

Qwen3.6-35B-A3B-abliterated-v4-GGUF

GGUF quantizations of Bahushruth/Qwen3.6-35B-A3B-abliterated-v4 for llama.cpp, Ollama, LM Studio, KoboldCPP, and other GGUF-compatible runtimes.

Uncensored model — refusal behavior removed via norm-preserving abliteration (0% refusal, full capability preservation). See the base model card for method details.

Blog post: Abliteration: Uncensoring LLMs via Weight Surgery

Quantizations

All standard quants are compatible with Ollama, LM Studio, KoboldCPP, and llama.cpp out of the box — no special flags needed.

FileSizeRAM RequiredNotes
...-BF16.gguf69 GB80+ GBFull precision
...-Q8_0.gguf37 GB48+ GBNear-lossless
...-Q6_K.gguf29 GB40+ GBVery high quality
...-Q5_K_M.gguf25 GB32+ GBRecommended for 48GB systems
...-Q4_K_M.gguf21 GB24+ GBGood quality, fits 24GB
...-IQ4_XS.gguf19 GB24+ GBHigh quality 4-bit (imatrix)
...-IQ4_NL.gguf20 GB24+ GBNon-linear 4-bit (imatrix)
...-Q3_K_M.gguf17 GB20+ GBGood for 16GB VRAM GPUs
...-IQ3_M.gguf15 GB20+ GBHigh quality 3-bit (imatrix)
...-IQ3_XXS.gguf14 GB16+ GBSmallest usable 3-bit (imatrix)
...-Q2_K.gguf13 GB16+ GB2-bit, quality tradeoffs
...-IQ2_M.gguf12 GB16+ GBSmallest, significant quality loss

MTP (Multi-Token Prediction)

Qwen3.6-35B-A3B includes an MTP draft head (blk.40) for speculative decoding. All standard quants above exclude MTP for maximum compatibility.

A separate BF16-MTP file is available for advanced users running llama-server directly:

FileSizeNotes
...-BF16-MTP.gguf~71 GBBF16 with MTP draft head included

To use MTP speculative decoding:

bash
./llama-server -m Qwen3.6-35B-A3B-abliterated-v4-BF16-MTP.gguf \
  --jinja --spec-type draft-mtp --spec-draft-n-max 1 -ngl 99
Runtime compatibility: MTP requires llama-server b9180+. Ollama does not support MTP yet. KoboldCPP 1.116.1+ and current LM Studio handle MTP fine, but many users are on older installs — the standard no-MTP quants are the safe default.

Quantization Types Explained

K-quants use a block-wise quantization scheme where weights are grouped into blocks and each block gets its own scale factor, preserving more precision than naive round-to-nearest quantization.

SuffixMeaning
Q8_08-bit uniform quantization (simplest, largest)
Q6_K6-bit k-quant. All tensor blocks use 6-bit.
Q5_K_M5-bit k-quant, medium variant. Important tensors bumped to Q6_K.
Q4_K_M4-bit k-quant, medium variant. Important tensors kept at Q5_K.
Q3_K_M3-bit k-quant, medium variant. Important tensors at Q4_K.
Q2_K2-bit k-quant. Aggressive compression with quality tradeoffs.
IQ4_XS4-bit importance-matrix quant. Uses imatrix calibration for better quality at same size.
IQ4_NL4-bit non-linear quant. Non-uniform quantization levels tuned to weight distributions.
IQ3_M3-bit importance-matrix quant. Medium quality.
IQ3_XXS3-bit importance-matrix quant. Extra small — minimum viable 3-bit.
IQ2_M2-bit importance-matrix quant. Extreme compression.

The _M (medium) suffix means a mixed-precision strategy: less important layers get the headline bit-width while critical layers (attention output, embeddings) get one level higher. This gives noticeably better quality than pure _S (small) variants at only ~5-10% size increase.

IQ quants use importance-matrix calibration (computed from wikitext-2) to allocate more precision to weights that matter most for model output quality. They achieve better perplexity than K-quants at the same file size.

Quickstart — Ollama

bash
# Recommended for Apple Silicon 48GB+ (M4 Pro, M4 Max, etc.)
ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:Q5_K_M

# For 24GB systems
ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:Q4_K_M

# For 10GB VRAM (RTX 3080)
ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:IQ3_XXS

Usage — llama.cpp

bash
huggingface-cli download Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF \
  Qwen3.6-35B-A3B-abliterated-v4-Q5_K_M.gguf --local-dir .

# Basic usage
./llama-cli -m Qwen3.6-35B-A3B-abliterated-v4-Q5_K_M.gguf \
  -p "You are a helpful assistant." \
  --chat-template chatml -cnv

Important Notes

  • —This is a MoE model (256 experts, 8 active per token). Despite 35B total params, only ~3B are active — efficient for its capability level.
  • —All standard quants work with Ollama, LM Studio, KoboldCPP, and llama.cpp without any special flags.
  • —IQ quants (IQ4XS, IQ3M, etc.) use imatrix calibration for better quality-per-bit.
  • —Tested and confirmed working on M4 Pro 48GB with Q5KM, and RTX 3080 10GB with IQ3_XXS.

Conversion Details

  • —Converter: llama.cpp/convert_hf_to_gguf.py --no-mtp
  • —Quantizer: llama-quantize
  • —imatrix: wikitext-2, 100 chunks, 16 threads
  • —Infrastructure: Modal (CPU-only, 16 cores, 128GB RAM)

Disclaimer

This model has had safety guardrails removed. Released for research purposes. The creator assumes no responsibility for downstream use.