Modelpublic

SeanWang0027/opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1

sourceHugging Faceapache-2.0updated 28d agoView on Hugging Face

0likes175downloads

Model Card

opdpolaris15Kqwen3-4bfromqwen3-30b-a3btopk16bf16epoch_1

On-Policy Distillation. Qwen3-4B trained against top-16 logprobs from Qwen3-30B-A3B-Thinking-2507, which scores and never generates: it runs one forward pass over what the student already wrote and reads the teacher's distribution at the states the student visited.

There is no teacher server and no GPU split — the teacher is a reward-model worker in the same Ray cluster, sharing every GPU with the actor.

Configuration


student	`Qwen/Qwen3-4B` — trained
teacher	`Qwen/Qwen3-30B-A3B-Thinking-2507` — MoE, ~3.4B active/token, scores only
data	`SeanWang0027/polaris_hard`, 15,368 competition-maths problems, 1 epoch
shapes	prompt ≤ 2048, response ≤ 7168, `ppo_max_token_len_per_gpu` 32768
OPD	`token_reward_direct`, top-16, `only_stu`, `student_p`
optimizer	AdamW, lr 1e-6 constant, wd 0.01, clip 1.0
dtype	bfloat16 — a deliberate deviation, see below
steps	240 (one epoch), `global_step_240`
hardware	8 × NVIDIA GH200 on TACC Vista, actor and teacher sharing all 8
wall clock	11 h 04 m, ~166 s/step

Code: `cl-from-nothing/online-rose` docs/POLARIS_OPD_QWEN3_30B_A3B_TEACHER.md.

The dtype deviation

The document specifies fp32 (upstream OPD's default). This run is bf16. It is not a preference: the teacher's weights are materialised in full host RAM on the coordinate-0 rank before FSDP shards them, and 30.53B × 4 B = 122 GB does not fit in a Vista node's 117.7 GiB of real Grace DRAM — the run is killed by the OOM killer, not by a CUDA error. The launcher's own header prescribes exactly this remedy below ~200 GB of node RAM. bf16 halves the teacher to 61 GB, which fits.

Nothing here is comparable to an fp32 OPD number.

What the run measured

The paper's condition (i) holds, and this is the part worth having. The doc states that whether a 3.4B-active MoE shares thinking patterns with a 4B student "has not been measured by anyone". It was measured here, across all 240 steps:

	step 1	step 120	step 240
`val-topk/overlap_ratio`	0.694	0.691	0.673
`val-topk/student_p_sum_intersection`	0.9982	0.9986	0.9985
`val-topk/teacher_p_sum_intersection`	0.9974	—	—

99.8% of the probability mass sits on the shared token set, steadily, for the whole run — above the 97–99% the paper reports for successful OPD. overlap_ratio drifts from 0.694 to 0.673, i.e. the top-16 sets overlap slightly less by the end, but the mass does not move: the disagreement is confined to tokens carrying almost no probability.

The student barely moved. Measured against Qwen/Qwen3-4B:


parameters that changed at all	1.66% (73.1 M / 4.41 B)
global relative L2 drift	6.58e-4
largest per-tensor drift	1.33e-3 (`layers.33.self_attn.k_proj`)
`critic/score/mean`	−0.208 (step 1) → −0.203 (step 240), flat throughout
`actor/entropy`	0.346 → 0.313
`actor/grad_norm`	2.88 → 1.50

For context, two ROSE runs on the same corpus and cluster, also bf16 but at lr 1e-5, moved 12.3% and 15.5% of their parameters. This run used lr 1e-6. A single AdamW step at 1e-6 is roughly 40× smaller than a bf16 half-ULP at |w| ≈ 0.02 (~3.9e-5), so most weights cannot represent the update at all. That is a consistent reading of three runs, not a controlled experiment — the students, teachers and objectives all differ, and no fp32 arm has been run.

No downstream evaluation has been run on this checkpoint or on the base model.

Treat this as a reproduction artifact of the training run and a measurement of OPD's condition (i) with a MoE teacher — not a model claimed to be better than Qwen/Qwen3-4B.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "SeanWang0027/opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")