yuhan-nlp/verl-grpo-medium-qwen3-4b-step129

verl-GRPO CollabLLM medium — Qwen3-4B, step 129

Qwen/Qwen3-4B trained with GRPO on the CollabLLM medium document-writing task using the verl CollabLLM recipe (no SFT warm start). This is a merged HF checkpoint — load it directly, no adapter step.

This checkpoint: the final checkpoint, kept to document the regression. BLEU is flat vs step 100 (t=-0.02) while interactivity drops significantly (t=-2.62).

Benchmark (medium, n=100)

Three roles, thinking off everywhere: Qwen3-4B assistant (this model), Qwen/Qwen3.5-9B user simulator, Qwen/Qwen3.5-27B judge. Flags: --user_sim_prompt sim_gap_paper_lazy_full --assistant_max_tokens 4096 --max_new_turns 14.

⚠️ Read before quoting these numbers

1. 1.Training peaked at step 100 and then regressed. Step 129 bought no BLEU and cost interactivity. Never select a checkpoint on verl's `critic/rewards/mean` — it rises through step 129, because the reward credits brevity directly while the benchmark judge penalises the interactivity that brevity costs.
2. 2.Prompting alone still wins on MR (1.1236 for proact vs 1.0761 for step 100) and needs no training. Step 100 wins on BLEU. Say which metric you mean.
3. 3.Every arm's MR gain is dominated by the token term, not document quality — so MR overstates quality improvements for all of them.
4. 4.⚠️ 35/100 train/eval prompt overlap with byte-identical reward targets. Because the BLEU reward target is the test reference on those rows, this is label leakage through the reward. See the dataset card.

Training setup

GRPO, train_batch_size=16, rollout.n=8, lr=1e-6, kl_loss_coef=0.001 (low-var KL), 8 GPUs (FSDP, world_size_8), reward = bleu_score (+1) + interactivity (+1) + token_amount (−0.1). Reference run: wandb photon/verlxcollabllm/e1adivlo, val@0 = 1.0618.

Full setup, the self-contained docker image, and the two acceptance tests are documented in studynotes/new_cluster_onboarding.md of the code repo.