Modelpublic

huluhuluu/qwen3-4b-instruct-2507-eagle3-sharegpt-full-context-epoch1-step40000

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes26downloads

Model Card

Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT checkpoints

Online EAGLE3 draft-model training run with SpecForge for Qwen/Qwen3-4B-Instruct-2507. This archive contains 47 checkpoints from epoch_0_step_5000 through epoch_9_step_231810. Each checkpoint is published as a separate Hub model repository in the companion collection.

This is a speculative-decoding draft model, not a standalone chat model. Pair it with the exact target model family.

Training parameters

Parameter	Value
Target model	`Qwen/Qwen3-4B-Instruct-2507`
Training method	Online EAGLE3 / SpecForge
Training data	Cleaned ShareGPT JSONL (local source; revision not recorded)
Epochs	`10`
Total optimizer steps	`231810`
Per-device batch size	`1`
Data-parallel size	`4`
Effective global batch size	`4`
Gradient accumulation	`1`
Learning rate	`1e-4`
LR schedule	Linear warmup (`1.5%`) then cosine annealing
Weight decay	`0.0`
Max gradient norm	`0.5`
Maximum sequence length	`2048`
EAGLE3 TTT length	`7`
Draft attention	`sdpa`
Target backend	SGLang (`flashinfer`)
Tensor parallel size	`1`
Seed	`0`
Save / log interval	`5000` / `50` steps
Dataset / dataloader workers	`64` / `4`

Architecture

LlamaForCausalLMEagle3, one decoder layer, hidden size 2560, intermediate size 9728, 32 attention heads, 8 key/value heads, draft vocabulary size 32000, target vocabulary size 151936, bfloat16 weights. The standard run does not set a sliding-window limit.

Checkpoint files

Every checkpoint repository contains model.safetensors, config.json, and training_state.pt. The latter stores optimizer/scheduler state and training arguments for resuming and should only be deserialized in a trusted environment. Prefer model.safetensors for inference.

Usage

Use a checkpoint repository as the SGLang speculative draft path, for example:

bash

python -m sglang.launch_server \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path huluhuluu/<checkpoint-repository> \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

The tree settings are starting values and should be benchmarked for the serving workload. No evaluation or safety metrics were recorded for this run.