Modelpublic

huluhuluu/qwen3-1.7b-eagle3-stage2-adapter-w4-bn1024-lr3e-4-epoch3-step160000

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes22downloads

Model Card

Qwen3-1.7B EAGLE3 Stage2 Adapter W4 BN1024 LR3e-4

This repository contains one SpecForge EAGLE3 draft-model checkpoint for use with Qwen/Qwen3-1.7B. It is a draft model for speculative decoding, not a standalone target language model.

Checkpoint

Field	Value
Source run	`qwen3-1.7b-eagle3-stage2-adapter-w4-bn1024-lr3e-4`
Checkpoint	`epoch_3_step_160000`
Epoch	`3`
Global step	`160000`
Files	`config.json`, `model.safetensors`, `training_state.pt`

Training Parameters

Parameter	Value
Base model	`Qwen/Qwen3-1.7B`
Method	SpecForge EAGLE3 online training with stage2 adapter
Dataset	ShareGPT-format SpecForge training set
Learning rate	`3e-4`
Batch size	`1`
Target batch size	`1`
Epochs configured	`8`
Max length	`2048`
Warmup ratio	`0.015`
Max grad norm	`0.5`
TTT length	`7`
Save interval	`5000`
Eval interval	`5000`
Seed	`0`
TP / DP size	`1 / 2`
Attention backend	`sdpa`
Target model backend	`sglang`
SGLang attention backend	`flashinfer`
Dataset build workers	`64`

Draft Model Configuration

Field	Value
Architecture	`LlamaForCausalLMEagle3`
dtype	`bfloat16`
Hidden size	`2048`
Intermediate size	`6144`
Draft layers	`1`
Attention heads	`16`
KV heads	`8`
Draft vocab size	`32000`
Vocab size	`151936`
Max position embeddings	`40960`
Stage2 adapter	enabled
Stage2 adapter bottleneck	`1024`
Stage2 max depth	`7`
Stage2 depth power	`2.0`

Notes

—This run uses a larger stage2 adapter bottleneck and higher learning rate than the W4 baseline.
—The checkpoint is intended to be loaded by SpecForge/EAGLE3-compatible code.
—training_state.pt is included for provenance and training-state inspection.
—No benchmark claim is made in this card.