jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller

Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller

A custom hybrid quantization of the Qwen3.8-27B base model, specifically designed to fit Multi-Token Prediction (MTP) and long context into a strict 16GB VRAM hardware budget (like consumer RTX 4080 / RTX 5080 cards).

How It Works

Instead of compressing the whole model equally, it is split by layer type:

• —Attention Layers (Brain & Logic): Kept high at `IQ4_XS` to protect reasoning, coding, and formatting logic.
• —FFN Layers: Compressed down to `IQ3_S` to trim file size. The trade-off is a minor loss in general knowledge and long-context recall, so if you use the model for something like creative writing, you'd probably be better off just using normal IQ4_XS quants.
• —The Result: The model file drops down to ~13.5 GB, leaving enough room for MTP and/or higher context size. The 5080 laptop GPU it was tested on (approximately 5070 desktop equivalent) results in 64k context with MTP at 50 t/s and 128k without MTP at around 30 t/s.

How It Was Made

The model was compiled using mainline llama.cpp.

The Compilation Command:

/home/llama.cpp/build/bin/llama-quantize \
  --imatrix Qwen3.8-27B.imatrix.mradermacher-ubergarm.dat \
  --tensor-type ffn_down=iq3_s \
  --tensor-type ffn_up=iq3_s \
  --tensor-type ffn_gate=iq3_s \
  BF16/Qwen3.8-27B-BF16.gguf \
  Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf \
  IQ4_XS \
  64

Credits & Acknowledgments

• —lemonyins — For the original methodology: https://huggingface.co/lemonyins/Qwen3.6-27B-uncensored-abliterated-i1-IQ4_XS-GGUF-Smaller.
• —cHunter789 — For providing the iMatrix data: https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4KSKT-GGUF
• —Unsloth — For the base bf16 model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
• —llama.cpp — the ggml/llama.cpp team for the base quantization framework.