Modelpublic

huluhuluu/qwen3-1.7b-eagle3-stage2-adapter-w4-learnable-scale-fp16-bn1024-lr1e-4-epoch7-step187120

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes19downloads

Model Card

Qwen3-1.7B EAGLE3 Stage2 Adapter W4 Learnable Scale FP16 BN1024

This repository contains one SpecForge EAGLE3 draft-model checkpoint for use with Qwen/Qwen3-1.7B. It is a draft model for speculative decoding, not a standalone target language model.

Checkpoint

Field	Value
Source run	`qwen3-1.7b-eagle3-stage2-adapter-w4-learnable-scale-fp16-bn1024-lr1e-4`
Checkpoint	`epoch_7_step_187120`
Epoch	`7`
Global step	`187120`
Files	`config.json`, `model.safetensors`, `training_state.pt`

Training Parameters

Parameter	Value
Base model	`Qwen/Qwen3-1.7B`
Method	SpecForge EAGLE3 online training with stage2 adapter
Dataset	ShareGPT-format SpecForge training set
Learning rate	`1e-4`
Batch size	`1`
Target batch size	`1`
Epochs configured	`8`
Max length	`2048`
Warmup ratio	`0.015`
Max grad norm	`0.5`
TTT length	`7`
Save interval	`5000`
Eval interval	`5000`
Seed	`0`
TP / DP size	`1 / 4`
Attention backend	`sdpa`
Target model backend	`sglang`
SGLang attention backend	`flashinfer`
Dataset build workers	`64`

Draft Model Configuration

Field	Value
Architecture	`LlamaForCausalLMEagle3`
dtype	`float16`
Hidden size	`2048`
Intermediate size	`6144`
Draft layers	`1`
Attention heads	`16`
KV heads	`8`
Draft vocab size	`32000`
Vocab size	`151936`
Max position embeddings	`40960`
Stage2 adapter	enabled
Stage2 adapter bottleneck	`1024`
Stage2 max depth	`7`
Stage2 depth power	`2.0`

Notes

—This run uses FP16 draft weights, a 1024 bottleneck, and learnable-scale stage2 adapter settings.
—The checkpoint is intended to be loaded by SpecForge/EAGLE3-compatible code.
—training_state.pt is included for provenance and training-state inspection.
—No benchmark claim is made in this card.