GavinLucky/SGLang-EAGLE3-Qwen3-0.6B-SpecForge

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/63993d721fad4d6eb265d999/X8LJWHUxZQxjUuLjGecR7.png" alt="specforge_team" width="60%"> </p>

EAGLE3 For Qwen3-0.6B

Available On

<p> <a href="https://huggingface.co/GavinLucky/SGLang-EAGLE3-Qwen3-0.6B-SpecForge"> <img src="https://img.shields.io/badge/🤗%20HuggingFace-GavinLucky%2FSGLang--EAGLE3--Qwen3--0.6B--SpecForge-yellow.svg?style=for-the-badge" alt="Hugging Face"> </a> <a href="https://www.modelscope.cn/models/gavinharris/SGLang-EAGLE3-Qwen3-0.6B-SpecForge"> <img src="https://img.shields.io/badge/🤖%20ModelScope-gavinharris%2FSGLang--EAGLE3--Qwen3--0.6B--SpecForge-blue.svg?style=for-the-badge" alt="ModelScope"> </a> <a href="https://huggingface.co/collections/GavinLucky/specbundle"> <img src="https://img.shields.io/badge/📦%20SpecBundle%20Collection-orange.svg?style=for-the-badge" alt="SpecBundle Collection"> </a> </p>

About

SpecBundle is an open-source initiative, jointly driven by the community and industry, to democratize speculative decoding by providing high-performance speculative decoding draft weights for mainstream open-source models.

This checkpoint was trained by the **SpecForge** Team and released as part of SpecBundle. The dataset is Aeala/ShareGPT_Vicuna_unfiltered (no target-model regeneration was used), and the draft model was trained on 68K samples for 10 epochs. This checkpoint was trained using the **SpecForge** framework.

中文版说明请见 README_zh.md。

Usage

You can use this checkpoint with the command below.

export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
python3 -m sglang.launch_server \
    --model Qwen/Qwen3-0.6B \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path GavinLucky/SGLang-EAGLE3-Qwen3-0.6B-SpecForge \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --mem-fraction-static 0.75 \
    --cuda-graph-max-bs 1 \
    --tp 1 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --dtype bfloat16

Performance

This checkpoint exhibits superior performance on various benchmarks.

<table> <tr> <th>Throughput</th> <th>Acceptance Length</th> </tr> <tr> <td><img src="./images/qwen30.6bthroughput.png" alt="throughput" /></td> <td><img src="./images/qwen30.6bacceptlength.png" alt="acceptancelength" /></td> </tr> </table>

You can reproduce the performance with the command below:

# clone specforge
git clone https://github.com/sgl-project/SpecForge.git
cd SpecForge/benchmarks
# run benchmarks
python bench_eagle3.py \
        --model Qwen/Qwen3-0.6B \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path GavinLucky/SGLang-EAGLE3-Qwen3-0.6B-SpecForge \
        --port 30004 \
        --config-list 8,0,0,0 8,3,1,4 8,5,1,6 8,5,3,6 8,7,1,8 8,7,4,8 \
        --benchmark-list gsm8k math500 mtbench humaneval livecodebench financeqa gpqa  \
        --dtype bfloat16 \
        --tp 1 \
        --name Qwen3-0.6B-specbundle

Acknowledgement

We sincerely appreciate the collective efforts from both the developers in the open-source community and our industrial partners, especially Ant Group AQ Team, Meituan, Nex-AGI (Qiji Zhifeng), EigenAI for their invaluable contributions to the release of SpecBundle Phase 1.