Modelpublic

huluhuluu/qwen3-4b-instruct-2507-eagle3-sharegpt-lr5e-5-epoch0-step10000

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes27downloads

Model Card

Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT (LR 5e-5)

Online EAGLE3 draft-model training run with SpecForge using a peak learning rate of 5e-5. This archive contains the two available checkpoints, epoch_0_step_5000 and epoch_0_step_10000; each is published as a separate Hub model repository in the companion collection.

This is a speculative-decoding draft model, not a standalone chat model. Pair it with the exact target model family.

Training parameters

Parameter	Value
Target model	`Qwen/Qwen3-4B-Instruct-2507`
Training method	Online EAGLE3 / SpecForge
Training data	Cleaned ShareGPT JSONL (local source; revision not recorded)
Epochs	`10` (run stopped at step `10000`)
Per-device batch size	`1`
Data-parallel size	`4`
Effective global batch size	`4`
Gradient accumulation	`1`
Learning rate	`5e-5`
LR schedule	Linear warmup (`1.5%`) then cosine annealing
Weight decay	`0.0`
Max gradient norm	`0.5`
Maximum sequence length	`2048`
EAGLE3 TTT length	`7`
Draft attention	`sdpa`
Target backend	SGLang (`flashinfer`)
Tensor parallel size	`1`
Seed	`0`
Save / log interval	`5000` / `50` steps
Dataset / dataloader workers	`64` / `4`

Architecture

LlamaForCausalLMEagle3, one decoder layer, hidden size 2560, intermediate size 9728, 32 attention heads, 8 key/value heads, draft vocabulary size 32000, target vocabulary size 151936, bfloat16 weights. The standard run does not set a sliding-window limit.

Checkpoint files

Every checkpoint repository contains model.safetensors, config.json, and training_state.pt. The latter stores optimizer/scheduler state and training arguments for resuming and should only be deserialized in a trusted environment. Prefer model.safetensors for inference.

Usage

Use a checkpoint repository as the SGLang speculative draft path with Qwen/Qwen3-4B-Instruct-2507 and the EAGLE3 speculative-decoding settings supported by your SGLang version. Tree settings should be benchmarked for the serving workload. No evaluation or safety metrics were recorded for this run.