barozp/Qwen3.8-27B-Opus-Distill-GGUF

Qwen3.8-27B-Opus-Distill-GGUF

GGUF quantizations of [barozp/Qwen3.8-27B-Opus-Distill](https://huggingface.co/barozp/Qwen3.8-27B-Opus-Distill) — a Qwen3.8-27B fine-tuned with LoRA on Claude Opus reasoning traces (merged), with the native vision tower and native MTP head carried over untouched.

Highlights

• —Reasoning-distilled, not just quantized. The LoRA was trained on 14,250 Opus chain-of-thought traces and merged into the base weights. Quantization only converts the weights — the reasoning gains travel with them unchanged.
• —Full multimodal. Native vision tower ships as a separate mmproj file (~0.9 GB). Text-only users can ignore it entirely.
• —Native MTP for self-speculative decoding. The model was released with its MTP head trained in — unlike grafted MTP setups, no approximation involved. Free speedups on compute-bound hardware.
• —imatrix-calibrated. All quants below Q3KM use an importance matrix built from the model's own reasoning-distillation data (see Imatrix).

Known issues

Reasoning loop under stacked output-format constraints. Reported by zxbc2023 (full writeup, discussion #1). Combining "no prose" with a second output-format constraint (e.g. "no markdown" or "no comments") can send this model into a non-converging self-verification reasoning loop -- it burns the entire token budget with zero visible output. Fully deterministic and reproducible at temp=0. Root cause: traced to part of the training data being sourced from reconstructed (not verbatim) Opus reasoning traces, not a capability gap.

Fixed in [barozp/Qwen3.8-27B-Opus-Distill-v2](https://huggingface.co/barozp/Qwen3.8-27B-Opus-Distill-v2) -- retrained on a rebuilt dataset where every row is traced to a verified genuine source. If you're hitting this, switch to v2.

Workaround if staying on this version: avoid combining "no prose" with another format constraint, or raise the generation token budget to >=4096 for constrained code-gen tasks.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (`--limit 500`). Base and distill ran with the identical harness, so the Δ column is the meaningful signal.

Reading the table:

• —Reasoning improved (ARC +4.2pt, GPQA +26pt), knowledge stayed flat (MMLU −0.001) and language modeling stayed flat (wikitext −0.09 ppl).
• —GPQA caveat: measured with thinking disabled (loglikelihood) — the base scores near random (25%) because it gets no chance to deliberate. The +26pt Δ is a valid same-protocol comparison, but do not compare 0.495 to Qwen's published 89.2 (measured with thinking ON, different harness).
• —ARC-Challenge is saturated for modern models; treat it as continuity with the Qwen3.6 release — GPQA is the stronger reasoning signal here.

Speed (MTP self-speculative decoding)

Not yet benchmarked for this exact model. On the Qwen3.6 sibling (same MTP mechanism, grafted there), measured with llama.cpp: +39% tok/s full offload, +67% partial offload with spec-decode ON. Native MTP (this model) is trained in and typically does at least as well. Guidance:

• —Compute-bound (full offload, strong GPU) → enable --spec-type draft-mtp.
• —Memory-bandwidth-bound (partial offload) → keep spec off.

Available quantizations

K-quants (Q8_0–Q3_K_M) are plain llama-quantize passes, no imatrix needed. IQ-quants (IQ3_XXS and below) require an importance matrix to run at all in current llama.cpp and are built from the one in this repo (see below).

Which one to pick:

• —Best quality with headroom → Q6_K or Q8_0
• —Best quality/size balance → Q4_K_M (default recommendation)
• —24 GB card → Q4KM; 16 GB card → Q3KM (partial offload)
• —Below that → IQ quants, accept the quality hit

Imatrix

imatrix.dat in this repo (512 samples from barozp/opus-reasoning-distill-train, context 512) was used to build the IQ quants above. It applies to any GGUF with this same architecture — including the base Qwen/Qwen3.8-27B — so it can be reused for re-quantization without recomputing it:

llama-quantize --imatrix imatrix.dat model-BF16.gguf model-IQ4_XS.gguf IQ4_XS

Note on `IQ1_M`: the MTP head (blk.64, the nextn.* decoder layer) is never exercised by a normal forward pass, so the imatrix has no data for it. llama-quantize pins that block to q4_K instead of failing, which is why IQ1_M lands at ~2.3 bits/weight (7.9 GB) rather than the ~1.8 a "pure" IQ1_M would suggest — the MTP head alone accounts for the difference, the rest of the model is quantized normally.

Vision (mmproj)

The vision tower is in Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf (~0.9 GB) in this repo. Load it alongside any quant for image/video input:

llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf

Text-only usage does not need mmproj and runs fine without it.

Quick start

# build llama.cpp with CUDA, then:

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on

Training details (source safetensors model)

• —Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• —Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• —LoRA targets: attention q/k/v/oproj on the 16 full-attention layers; FFN gate/up/downproj on all 64 layers (Gated-DeltaNet projections untouched)
• —Data: barozp/opus-reasoning-distill-train (14,250) + -validation (750, held out)
• —Run: 1 epoch (891 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~5h52m on A100 80GB
• —Final validation loss: 0.4647
• —Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained

Notes

• —Thinking mode is on by default (same as the base model). The GGUF embeds the chat template; how thinking is toggled depends on the llama.cpp version / frontend (e.g., LM Studio exposes the setting in its UI).
• —Conversion: llama.cpp convert_hf_to_gguf.py from the corrected multimodal config (nested text_config + vision_config).
• —No chaining: every quant was produced directly from the BF16 GGUF, so errors do not accumulate across the ladder.

Source chain

The full Qwen3.8-27B Opus Distill family:

• —Qwen/Qwen3.8-27B — base model
• —barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF — v2, GGUF quants
• —barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 — v2, FP8 build
• —barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit — v2, MLX 4-bit
• —barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit — v2, MLX 8-bit
• —barozp/Qwen3.8-27B-Opus-Distill-v2 — v2, BF16 weights
• —barozp/Qwen3.8-27B-Opus-Distill-GGUF — v1, GGUF quants
• —barozp/Qwen3.8-27B-Opus-Distill — v1, BF16 weights

This release: v1, GGUF quants (the card you are reading).