inference-optimization/Qwen3-Next-80B-A3B-Instruct-GSM8K-MTP-finetuned

Qwen3-Next-80B-A3B-Instruct — GSM8K MTP Finetuned (Epoch 0)

This is Qwen/Qwen3-Next-80B-A3B-Instruct with its native Multi-Token Prediction (MTP) head replaced by a version finetuned on GSM8K math reasoning data to improve speculative decoding acceptance rates.

What changed

The MTP head weights (mtp.* keys) have been replaced with weights finetuned using the speculators library. All other model weights (transformer layers, embeddings, lm_head) are identical to the original Qwen3-Next-80B release.

Training details

Acceptance rate results

Evaluated on RedHatAI/speculator_benchmarks:math_reasoning.jsonl with num_speculative_tokens=3:

+22.5% improvement in mean accepted tokens over the base MTP head.

Usage

Load directly with vLLM — the MTP head is embedded in the model weights:

from vllm import LLM, SamplingParams

llm = LLM(
    model="inference-optimization/Qwen3-Next-80B-A3B-Instruct-GSM8K-MTP-finetuned",
    tokenizer_mode="auto",
    tensor_parallel_size=4,
    gpu_memory_utilization=0.8,
    speculative_config={
        "method": "mtp",
        "num_speculative_tokens": 3,
    },
    enable_chunked_prefill=False,
)

sampling_params = SamplingParams(temperature=0.6, top_p=0.95)
outputs = llm.generate(["Solve: Janet has 3 apples..."], sampling_params)

Generation pipeline

This checkpoint was produced using the speculators library via the following steps:

1. 1.Data generation — hidden states extracted from Qwen3-Next-80B-A3B-Instruct on GSM8K using examples/fast_mtp/generate_dataset.py
2. 2.Finetuning — MTP head trained with examples/fast_mtp/04_finetune.py
3. 3.Weight stitching — finetuned MTP weights stitched back into the verifier using examples/fast_mtp/stitch_weights.py