Modelpublic

huluhuluu/qwen3-4b-instruct-2507-eagle3-sharegpt-sw512-epoch4-step100000

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes24downloads

Model Card

Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT (SW512)

Online EAGLE3 draft-model training run with SpecForge and a draft sliding window of 512 tokens. This archive contains 47 checkpoints from epoch_0_step_5000 through epoch_9_step_231810; each checkpoint is published as a separate Hub model repository in the companion collection.

This is a speculative-decoding draft model, not a standalone chat model. Pair it with the exact target model family.

Training parameters

Parameter	Value
Target model	`Qwen/Qwen3-4B-Instruct-2507`
Training method	Online EAGLE3 / SpecForge
Training data	Cleaned ShareGPT JSONL (local source; revision not recorded)
Epochs	`10`
Total optimizer steps	`231810`
Per-device batch size	`1`
Data-parallel size	`4`
Effective global batch size	`4`
Gradient accumulation	`1`
Learning rate	`1e-4`
LR schedule	Linear warmup (`1.5%`) then cosine annealing
Weight decay	`0.0`
Max gradient norm	`0.5`
Maximum sequence length	`2048`
EAGLE3 TTT length	`7`
Draft sliding window	`512` tokens
Draft attention	`sdpa`
Target backend	SGLang (`flashinfer`)
Tensor parallel size	`1`
Seed	`0`
Save / log interval	`5000` / `50` steps
Dataset / dataloader workers	`64` / `4`

Architecture

LlamaForCausalLMEagle3, one decoder layer, hidden size 2560, intermediate size 9728, 32 attention heads, 8 key/value heads, draft vocabulary size 32000, target vocabulary size 151936, bfloat16 weights, and causal sliding-window attention of 512 tokens.

Checkpoint files

Every checkpoint repository contains model.safetensors, config.json, and training_state.pt. The latter stores optimizer/scheduler state and training arguments for resuming and should only be deserialized in a trusted environment. Prefer model.safetensors for inference.

Usage

Use a checkpoint repository as the SGLang speculative draft path with Qwen/Qwen3-4B-Instruct-2507 and the EAGLE3 speculative-decoding settings supported by your SGLang version. Tree settings should be benchmarked for the serving workload. No evaluation or safety metrics were recorded for this run.