samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled-GGUF

Qwen3-Coder-Next — Opus 4.6 Reasoning Distilled (GGUF)

GGUF quantizations of the full fine-tuned Qwen/Qwen3-Coder-Next (~80B total / ~3B active, MoE) with Claude Opus 4.6 reasoning distillation. Trained on 8x H100 80GB SXM with DeepSpeed ZeRO-3, all parameters.

Model Details

Available Quantizations

Benchmark Results

Evaluated by Claude Opus 4.6 across 26 tests in 6 categories, scored 1-10 on correctness, completeness, clarity, and adherence to instructions.

Detailed Test Results

Coding (6 tests)

Bug Detection (5 tests)

Probability (5 tests)

Tool Calling (5 tests) — Largest improvement

Logic (2 tests)

Instruction Following (3 tests)

Key Findings

• —Tool Calling: Largest improvement (+3.8). Base model outputs only the first tool call and stops. Opus Distilled plans full multi-step tool chains with reasoning between steps.
• —Bug Detection: Opus Distilled provides more structured analysis with severity tables, timeline diagrams, and catches more edge cases (+0.6).
• —Coding: Opus Distilled favors class-based architectures with better design patterns. Caught a SQL bug (window function in WHERE clause) that Base missed.
• —Probability: Base is more concise and made fewer computation errors. Opus Distilled made an error on a Markov Chain steady-state calculation.
• —Logic: Base makes better progress within token budgets — Opus Distilled spends more tokens on preamble.
• —Instruction Following: Base adheres more strictly to output format constraints (e.g., "output ONLY valid JSON").

Verdict

Opus-Distilled wins overall driven by massively better tool calling and slightly better bug detection and coding. Base wins on math/probability (fewer errors), logic (better token efficiency), and instruction following (better constraint adherence). For coding assistant use cases where tool calling matters, Opus-Distilled is clearly superior.

Performance

Both models run at comparable speeds on RTX PRO 6000 Blackwell (96GB) with Q8_0:

Usage with llama.cpp

Basic Serving

llama-server \
  --model Qwen3-Coder-Next-Opus-Distilled-Q8_0.gguf \
  --n-gpu-layers -1 \
  --ctx-size 262144 \
  --host 0.0.0.0 --port 8081

With Reasoning Support (Recommended)

The model produces <think>...</think> reasoning blocks. To properly separate these from the visible output, use a custom chat template with --reasoning-format deepseek:

llama-server \
  --model Qwen3-Coder-Next-Opus-Distilled-Q8_0.gguf \
  --n-gpu-layers -1 \
  --ctx-size 262144 \
  --chat-template-file qwen3-think.jinja \
  --reasoning-format deepseek \
  --host 0.0.0.0 --port 8081

This puts the thinking in message.reasoning_content and keeps message.content clean.

Chat Template

The repo includes a ready-to-use `qwen3-think.jinja` chat template that supports both <think> reasoning and Qwen3's XML tool calling format. Download it alongside the GGUF:

# Download model + template
huggingface-cli download samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled-GGUF \
  Qwen3-Coder-Next-Opus-Distilled-Q8_0.gguf qwen3-think.jinja \
  --local-dir ./

Note: The base Qwen3-Coder-Next chat template does NOT include <think> tag support. You must use the provided qwen3-think.jinja template for proper reasoning and tool calling behavior. Thanks to @codyknowscode for the tool calling fix.

OpenAI-Compatible API

The model serves an OpenAI-compatible API. Example request:

curl http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "opus-distilled",
    "messages": [{"role": "user", "content": "Implement a thread-safe LRU cache in Rust"}],
    "max_tokens": 4096,
    "temperature": 0.6
  }'

Response includes reasoning_content (thinking) separate from content (answer).

Training Details

Hardware & Infrastructure

• —GPUs: 8x NVIDIA H100 80GB SXM with NVLink
• —System RAM: 2 TB DDR5
• —Distribution: DeepSpeed ZeRO-3 (parameters sharded across all 8 GPUs)
• —Optimizer Offload: AdamW optimizer states offloaded to CPU RAM (~700GB)
• —Platform: RunPod

Hyperparameters

Training Progression

Datasets

3,204 examples after quality filtering (required <think> tags and >200 characters of assistant content):

Data Format

Each training example follows this structure:

<|im_start|>user
{problem}<|im_end|>
<|im_start|>assistant
<think>
{chain-of-thought reasoning}
</think>

{final answer}<|im_end|>

Quality Filter

Examples were filtered to require:

1. 1.At least one assistant message containing <think> tags
2. 2.Assistant content longer than 200 characters

This removed low-quality or non-reasoning examples from the combined dataset.

Reasoning Format

The model produces reasoning inside <think>...</think> tags:

<think>
Let me analyze this step by step...
1. First consideration
2. Second consideration
3. Conclusion
</think>

Here is the final answer based on my analysis.

HF Safetensors

For the full-precision HuggingFace model (BF16 safetensors), see samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled.