Modelpublic

RockToken/qwen3_30b_a3b_to_4b_onpolicy_5k_src30k-35k_cont

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes24downloads

Model Card

Qwen3-4B distilled from Qwen3-30B-A3B — On-policy 5k (continual, prompts 30k–35k)

Continual on-policy knowledge distillation of a Qwen3-4B student toward the Qwen3-30B-A3B (MoE) teacher, on 5,000 math prompts from openthoughts3_math_50k (positions 30k–35k, disjoint from prior training slices).

The student for this run is RockToken/qwen3-30b-a3b-to-4b-onpolicy-10k, so the effective training exposure of this checkpoint is:

1.Off-policy KD on 20k teacher responses (positions 0–20k of openthoughts3_math_50k)
2.On-policy KD on 10k prompts (positions 20k–30k) — produced the 10k model
3.On-policy KD (this run) on 5k prompts (positions 30k–35k) — this checkpoint

Stages 2 and 3 use the same on-policy pipeline; only the seed weights, prompt slice, and step count differ.

Models

Role	Model
Student	Qwen3-4B-Instruct-2507 (after stages 1–2 above)
Teacher	Qwen/Qwen3-30B-A3B-Instruct-2507 (MoE, ~3B active)

enable_thinking=False throughout (both stages).

Training data

—Source: openthoughts3_math_50k (math split of OpenThoughts-3)
—Slice used here: prompts 30k–35k → RockToken/openthoughts_prompt_math_5k_src30k-35k
—5,000 single-user-turn prompts, mean length 261 chars, p99 845 chars

Training setup

Framework: KDFlow — FSDP2 + SGLang rollout, Ray-orchestrated GPU co-location with sleep/wakeup.

Hardware: 1× node, 4× H100 (80 GB), ~34 h wall-clock (~136 GPU-hours).

Key hyperparameters (on-policy stage; same as stage 2)

Group	Value
Backend	fsdp2, bf16, gradient ckpt on
Epochs	1
Train batch	4 (micro 1)
Learning rate	2e-6, cosine, warmup 5%
KD ratio	1.0
KD loss	reverse KL (`rkl`)
KD algorithm	`vanilla_kd`
Temperature (KD)	1.0
Rollout engine	SGLang, TP=2, 1 engine
Rollout batch	2 prompts × 4 samples/prompt
`generate_max_len`	8000
`prompt_max_len`	800 (total `max_len` 8800)
Sampling	temperature 1.0, top-p 1.0
Teacher	TP=4, sleep/wakeup enabled

Off-policy stage 1 hyperparameters (for context): batch 128, LR 2e-5, kd_ratio 0.5, kd_loss KL, max_len 16384, packing_samples on, ring attention.

Intended use

Research on distillation dynamics. Domain skew is math (OpenThoughts-3 math split).

Limitations

—Trained end-to-end on math prompts only; not tuned for chat, safety, or non-math domains.
—enable_thinking=False — this student does not emit <think> traces.
—Continual training may drift capabilities relative to the base checkpoint; see the wandb curves for loss dynamics.

Weights & Biases

legendary-music-16 (stage 2, 10k) and lemon-sound-17 / 2gvd60dw (this run, stage 3) under project KDFlow, entity zhao666.