Modelpublic

seomh/Qwen3-8B-Base-OpenThoughts3Math-SFT-step750-lr5e-6

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face

0likes202downloads

Model Card

Qwen3-8B-Base -- OpenThoughts3 math SFT, step 750

Intermediate checkpoint (step 750 of a planned 2000), bf16 weights only. Not yet evaluated.


base	`Qwen/Qwen3-8B-Base`
data	`knowledge-distillation/openthoughts3_math`, 103,760 two-turn conversations
sequence length	32768 (truncates nothing: mean 13.8k, p99 16.8k, max 16.8k tokens)
global batch	255
optimizer	AdamW, lr 5e-6, cosine, 3% warmup, weight decay 0.01, grad clip 1.0
precision	bf16, FSDP over 3 GPUs, verl

Every assistant message in the data is a <think>...</think> trace ending in \boxed{}, and the model is trained to reproduce that. It uses a plain ChatML template with no thinking logic -- Qwen3's own template strips <think> from all but the last assistant turn, which would have removed the reasoning from the training targets. No enable_thinking flag applies.

A turn ends with <|im_end|> (151645); config.json and generation_config.json both say so. Qwen3-8B-Base's stock generation_config.json says 151643 instead, which yields a model that never terminates under vLLM. Generate with max_new_tokens around 16384.