barozp/Qwen3.8-27B-Opus-Distill

Qwen3.8-27B-Opus-Distill

Qwen/Qwen3.8-27B fine-tuned (LoRA, merged) on Claude Opus reasoning traces. Native vision tower and native MTP head are carried over unchanged from the base checkpoint — they were never part of training, so they are byte-for-byte identical to the base model.

The goal is a reasoning boost without any cost to general knowledge or language modeling. Measured: reasoning improves (ARC +4.2pt, GPQA +26pt), knowledge and language modeling stay flat. See the benchmark table below.

Known issues

Reasoning loop under stacked output-format constraints. Reported by zxbc2023 (full writeup, discussion #1). Combining "no prose" with a second output-format constraint (e.g. "no markdown" or "no comments") can send this model into a non-converging self-verification reasoning loop -- it burns the entire token budget with zero visible output. Fully deterministic and reproducible at temp=0. Root cause: traced to part of the training data being sourced from reconstructed (not verbatim) Opus reasoning traces, not a capability gap.

Fixed in [barozp/Qwen3.8-27B-Opus-Distill-v2](https://huggingface.co/barozp/Qwen3.8-27B-Opus-Distill-v2) -- retrained on a rebuilt dataset where every row is traced to a verified genuine source. If you're hitting this, switch to v2.

Workaround if staying on this version: avoid combining "no prose" with another format constraint, or raise the generation token budget to >=4096 for constrained code-gen tasks.

Model overview

• —Base model: Qwen/Qwen3.8-27B — dense 27B, native vision-language, native MTP.
• —Method: LoRA (r=64, alpha=64, dropout=0.05), merged into the base weights.
• —LoRA targets: attention q/k/v/o_proj on the 16 full-attention layers only; FFN gate/up/down_proj on all 64 layers. Gated-DeltaNet (linear-attention) projections are left untouched, matching the Qwen3.6 recipe.
• —Training data: `barozp/opus-reasoning-distill-train` (14,250 examples) + `-validation` (750, held out).
• —Training run: 1 epoch (891 steps), lr=1e-4 cosine with 3% warmup, effective batch 16 (micro-batch 1 × grad-accum 16), MAX_SEQ=4096, bf16, ~5h52m on an A100 80GB.
• —Final validation loss: 0.4647 (step 891/891).
• —Vision + MTP: carried over from base, never trained.

Highlights

• —Reasoning improves, nothing else degrades. ARC-Challenge and GPQA-Diamond both rise while MMLU (general knowledge) and wikitext (language modeling) stay flat within noise.
• —Reflex-level reasoning gets the biggest boost. GPQA is measured in loglikelihood mode with thinking disabled — the model has no chance to deliberate. Distillation more than doubles that "reflex" score, meaning the model internalized step-by-step reasoning rather than just learning to generate <think> blocks.
• —Vision and MTP are fully intact. This is a complete multimodal model with self-speculative decoding, not a text-only strip-down.

Benchmark results

Measured with lm-evaluation-harness, 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (`--limit 500`). Base and distill were run with the identical harness, so the Δ column is the meaningful signal — not the absolute values.

Important caveats

• —GPQA is not comparable to Qwen's published 89.2. Qwen reports GPQA with thinking mode ON and their own harness. Here GPQA is measured in loglikelihood mode with thinking OFF — the base model scores near random (25%) precisely because it gets no chance to reason. The +26pt Δ is a valid same-protocol comparison (base vs distill), but do not place 0.495 next to Qwen's 89.2 as if they were the same measurement.
• —ARC-Challenge is saturated for modern models. The +4.2pt Δ is real and consistent with the Qwen3.6 release, but ARC (2018, middle-school science) is not a headline reasoning benchmark anymore. GPQA is the stronger signal here.
• —QUICK mode limits each task to 500 samples; hellaswag and arc_challenge are noisy at that size. Treat small Δ values (±0.01) on those as noise.

Quick start

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "barozp/Qwen3.8-27B-Opus-Distill", dtype=torch.bfloat16, device_map="auto",
)
processor = AutoProcessor.from_pretrained("barozp/Qwen3.8-27B-Opus-Distill")

Text-only inference also works via AutoModelForCausalLM (vision is ignored, which is the expected/standard behavior for that class).

Related models

• —Qwen3.8-27B — the base model.
• —barozp/Qwen3.8-27B-Opus-Distill-LoRA-Adapter — the raw LoRA adapter (private; the merged weights here are the same thing applied).
• —barozp/Qwen3.8-27B-Opus-Distill-GGUF — GGUF quantizations, BF16 down to IQ1_M, imatrix-calibrated, includes the vision mmproj file.
• —barozp/Qwen3.8-27B-Opus-Distill-v2 — bugfix release, see Known Issues above.