rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled

Qwen3.8-27B-Claude-Opus-Reasoning-Distilled

A LoRA fine-tune of Qwen/Qwen3.8-27B distilled on reasoning traces from Claude Opus 4.6/4.7, targeting the deep, structured chain-of-thought style Opus is known for — mapped onto an open-weight dense 27B you can run and fine-tune yourself.

Trained text-only (vision tower untouched, not targeted by LoRA) with Unsloth on a single H100.

🔗 Quantized GGUF versions (Q8_0 → Q4_K_M) with verified MTP speculative decoding available at [rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled-GGUF](https://huggingface.co/rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled-GGUF).

🧠 What this is

Qwen3.8-27B is natively vision-language with a hybrid attention stack (16/64 full-attention layers, 48/64 Gated DeltaNet/linear-attention layers). This fine-tune targets only the text reasoning path — both attention types via LoRA, MLP layers, vision tower left frozen and unused.

• —Base model: Qwen/Qwen3.8-27B (dense, 27B, hybrid linear+full attention, MTP head)
• —Method: LoRA (r=32, alpha=32) via Unsloth, 4-bit QLoRA base
• —Response-only masking: loss computed only on assistant turns (train_on_responses_only)
• —Hardware: 1× NVIDIA H100 NVL (95GB), Vast.ai
• —Training data: ~21.5K examples combining real Opus extended-thinking traces and reconstructed-reasoning traces over genuine Opus outputs (see Dataset Composition below)

📊 Training Details

Status: pipeline-validation run, not a fully converged model. 150 steps at batch 18 covers ~12.6% of one epoch — enough to confirm the training pipeline (LoRA on hybrid attention, masking, chat template, checkpointing) works end-to-end, not enough for the model to have generalized across the full dataset. A full run (1+ epoch, ~1,194+ steps) is needed before this should be treated as a finished distillation. Loss trended downward over the run (high ~0.95-1.1 early, several sub-0.6 readings by step 70-90) but with a cosine schedule calibrated for exactly 150 steps — this checkpoint cannot simply be resumed for a longer run; a full run needs a fresh schedule sized to the full dataset.

📚 Dataset Composition

Transparency note: ~62% of the dataset (the two TraceInversion sets) has a genuine final answer from Opus but a reconstructed reasoning trace, not Opus's actual internal thinking. This is disclosed so downstream users know the reasoning style is a plausible approximation for roughly two-thirds of the data, not a verbatim mirror throughout. The lordx64 subset (~38%) is the closest to authentic Opus thinking style.

We deliberately excluded several other "Claude-distilled" datasets found on Hugging Face after inspection — some contained self-referential synthetic text describing itself as "exemplifying" a model's style rather than real model output, and others were low-effort duplicate repos farming downloads. Always verify teacher_model/usage provenance fields before trusting a distillation dataset.

📈 Base Model Benchmarks (Qwen/Qwen3.8-27B, not this fine-tune)

These are official Qwen3.8-27B numbers for the base model, included for reference — this fine-tune has not been independently re-benchmarked yet.

Text Performance

VL Performance (base model only — this fine-tune does not target vision)

Full base model benchmark tables: Qwen/Qwen3.8-27B model card.

🚀 Usage

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled",
    max_seq_length = 8192,
    dtype = None,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "Explain the difference between TCP and UDP."}]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
outputs = model.generate(inputs, max_new_tokens=1024, temperature=1.0, top_p=0.95, top_k=20)
print(tokenizer.decode(outputs[0]))

Thinking is on by default (<think>...</think> block before the final answer), matching the base Qwen3.8 chat template. Use reasoning_content as a separate message field if constructing multi-turn history manually — do not pack the thinking block into content, the chat template expects it separate.

⚠️ Known Limitations

• —This checkpoint is a pipeline-validation run (150 steps, ~12.6% of one epoch), not a fully converged fine-tune. Expect it to show the target <think> format and some stylistic shift, but not robust generalization across task domains. A full training run is needed for production use — see status note above.
• —Dataset partially composed of reconstructed (not captured) reasoning traces — see Dataset Composition above
• —Text-only fine-tune; vision tower is unmodified base weights, not evaluated

🙏 Acknowledgments

Training methodology based on the Jackrong fine-tuning guide. Thanks to lordx64 and Jackrong for the source reasoning datasets.

Maintained by rico03