Modelpublic

SeanWang0027/student_prefix_polaris_15K_olmo3-7b_continual_Q_qwen3-30b-a3b_cutoff4096_epoch_1_mask

sourceHugging Faceapache-2.0updated 29d agoView on Hugging Face

0likes326downloads

Model Card

studentprefixpolaris15Kolmo3-7bcontinualQqwen3-30b-a3bcutoff4096epoch1_mask

Online ROSE across a vocabulary boundary. Olmo-3-7B-Think-SFT (student, vocab 100278) trained on continuations written by Qwen3-30B-A3B-Thinking-2507 (teacher, vocab 151936), with cross-entropy applied only to the teacher's tokens.

prompt ---- student prefix (4096 tok) ------- teacher continuation (~941 tok) ----
            response_mask 0                   response_mask 1
            context, on-policy                the target, off-policy

The two vocabularies are different, so the boundary is crossed as text: the student's prefix is decoded, the teacher is shown it under its own chat template, and the reply is tokenised back into the student's vocabulary. The mask boundary is len(prefix_ids) by construction — the student's ids are never re-encoded.

Configuration


student	`allenai/Olmo-3-7B-Think-SFT` — 7.30B, vocab 100278, no GQA, 24 sliding / 8 full attention
teacher	`Qwen/Qwen3-30B-A3B-Thinking-2507` — MoE, vocab 151936, 2 vLLM TP=1 replicas
data	`SeanWang0027/polaris_hard`, 15,368 competition-maths problems, 1 epoch
cut	K = 4096 prefix + t = 1024 teacher tokens → `max_response_length` 5120
schedule	asynchronous, depth 3
batch	`train_batch_size` 64, `n` 4 → 256 rows per block
optimizer	AdamW, lr 1e-5 constant, no warmup, wd 0.01, clip 1.0, bf16
steps	240 (one epoch), `global_step_240`
hardware	10 × NVIDIA GH200 on TACC Vista — 2 teacher + 8 FSDP trainer ranks
wall clock	8 h 37 m, 129 s/step

Code: `cl-from-nothing/online-rose` @ 4264fe9. Every rollout of this run is published as `SeanWang0027/polaris_rose_rollouts_olmo3-7b_from_qwen3-30b-a3b_cutoff4096_240steps` — 61,440 rows with the teacher's actual continuation and the full token accounting.

What actually happened during training

The objective moved.


`actor/ce_nll_before`, mean of first 20 steps	0.6398
`actor/ce_nll_before`, mean of last 20 steps	0.5425 (−15.2%)
`actor/grad_norm`	1.10 → 0.20
`rose/pool_failures`	0 across all 240 steps (58,624 teacher calls)
`rose/teacher_hidden_frac`	1.000 — the teacher was never the bottleneck
`rose/seam_shift`	0 on all 61,440 rows
`rose/teacher_finished`	2.07%
`rose/empty`	4.17%

But 84.5% of the weights never moved a single bit. Measured against the base model:

	this run	a same-family control run
parameters that changed at all	15.46% (1.13 B / 7.30 B)	12.32%
global relative L2 drift	2.33e-3	1.75e-3
largest per-tensor drift	8.57e-3 (`layers.31.self_attn.q_proj`)	3.36e-3
`ce_nll_before` over the run	−15.2%	+0.3% (flat)

The control is a Qwen3-4B ← Qwen3-32B same-vocabulary run on the same corpus, K, t and schedule (checkpoint); the student and teacher differ, so it bounds nothing on its own — it is context, not a matched comparison.

This run moved measurably more than that one, in the direction the loss moved. The ceiling is the same, though, and it is worth stating: the launcher passes actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16, which overrides the torch.float32 verl otherwise forces for the actor — and verl's own comment at fsdp_workers.py:302 reads "we have to create model in fp32. Otherwise, the optimizer is in bf16, which is incorrect." With bf16 master weights and bf16 Adam moments, one step at lr = 1e-5 is the same order as a bf16 half-ULP at |w| ≈ 0.02, so most weights cannot move. The repo keeps bf16 deliberately, so that every number in it stays comparable; an fp32 arm is a separate control that has not been run.

No downstream evaluation has been run. Neither this checkpoint nor allenai/Olmo-3-7B-Think-SFT has been measured on AIME 2025 or anything else. The reward logged during training is log-only — compute_policy_loss_ce ignores advantages — and it scores a truncated prefix spliced to a partial continuation.

Treat this as a reproduction artifact of the training run. A falling CE on the teacher's tokens is not a claim about downstream ability. If you want that comparison, run both models under identical settings.

Two numbers that are properties of the configuration, not defects

—`rose/teacher_tokens` averages 940.6, not the configured 1024. The teacher generates 1024 tokens in its vocabulary; re-encoded into the student's they are fewer. Read the supervised length from the metric, never from t.
—`rose/teacher_finished` is 2.07%. After a 4096-token prefix, 1024 more tokens almost never reach the end of a competition-maths solution, so the student is overwhelmingly taught to continue and only rarely to stop.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "SeanWang0027/student_prefix_polaris_15K_olmo3-7b_continual_Q_qwen3-30b-a3b_cutoff4096_epoch_1_mask"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")