samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled
Qwen3-Coder-Next — Opus 4.6 Reasoning Distilled (Full Fine-Tune)
Full parameter fine-tune of Qwen/Qwen3-Coder-Next (~80B total / ~3B active, MoE) with Claude Opus 4.6 reasoning distillation. Trained on 8x NVIDIA H100 80GB SXM with DeepSpeed ZeRO-3.
Model Details
Benchmark Results
Evaluated by Claude Opus 4.6 across 26 tests in 6 categories, scored 1-10 on correctness, completeness, clarity, and adherence to instructions.
Detailed Test Results
Coding (6 tests)
Bug Detection (5 tests)
Probability (5 tests)
Tool Calling (5 tests) — Largest improvement
Logic (2 tests)
Instruction Following (3 tests)
Key Findings
- Tool Calling: Largest improvement (+3.8). Base model outputs only the first tool call and stops. Opus Distilled plans full multi-step tool chains with reasoning between steps.
- Bug Detection: Opus Distilled provides more structured analysis with severity tables, timeline diagrams, and catches more edge cases (+0.6).
- Coding: Opus Distilled favors class-based architectures with better design patterns. Caught a SQL bug (window function in WHERE clause) that Base missed.
- Probability: Base is more concise and made fewer computation errors. Opus Distilled made an error on a Markov Chain steady-state calculation.
- Logic: Base makes better progress within token budgets — Opus Distilled spends more tokens on preamble.
- Instruction Following: Base adheres more strictly to output format constraints (e.g., "output ONLY valid JSON").
Verdict
Opus-Distilled wins overall driven by massively better tool calling and slightly better bug detection and coding. Base wins on math/probability (fewer errors), logic (better token efficiency), and instruction following (better constraint adherence). For coding assistant use cases where tool calling matters, Opus-Distilled is clearly superior.
Performance
Both models run at comparable speeds on RTX PRO 6000 Blackwell (96GB):
Training Details
Hardware & Infrastructure
- GPUs: 8x NVIDIA H100 80GB SXM with NVLink
- System RAM: 2 TB DDR5
- Distribution: DeepSpeed ZeRO-3 (parameters sharded across all 8 GPUs)
- Optimizer Offload: AdamW optimizer states offloaded to CPU RAM (~700GB)
- Platform: RunPod
Hyperparameters
Training Progression
Datasets
3,204 examples after quality filtering (required <think> tags and >200 characters of assistant content):
Data Format
Each training example follows this structure:
<|im_start|>user
{problem}<|im_end|>
<|im_start|>assistant
<think>
{chain-of-thought reasoning}
</think>
{final answer}<|im_end|>Quality Filter
Examples were filtered to require:
- At least one assistant message containing
<think>tags - Assistant content longer than 200 characters
This removed low-quality or non-reasoning examples from the combined dataset.
Reasoning Format
The model produces reasoning inside <think>...</think> tags:
<think>
Let me analyze this step by step...
1. First consideration
2. Second consideration
3. Conclusion
</think>
Here is the final answer based on my analysis.When serving with llama.cpp, use --reasoning-format deepseek with a thinking-aware chat template to separate reasoning from visible output.
GGUF Quantizations
See samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled-GGUF for quantized GGUF versions (Q4KM, Q6K, Q80, BF16).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled",
torch_dtype="bfloat16",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
"samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled",
trust_remote_code=True,
)
messages = [{"role": "user", "content": "Implement a thread-safe LRU cache in Python"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))