varadsrivastava/lm-playschool-qwen3.5-2b-iter4

R4 — Corrective feedback (DPO)

Part of a five-regime developmental sweep of post-training methods for dialogue-game competence (LM Playschool Challenge 2026, team DAIR).

A second DPO round on the merged R2 model using 201 first-move preference pairs from two rollout passes (greedy and t=0.7): 107 on-policy pairs from instances where the two passes disagreed in outcome (chosen = the model's own success, rejected = its own failure on the identical instance) and 94 hybrid pairs from instances failed in both passes (chosen = a stronger model's success on that instance — a recast). Same hyperparameters as R2.

Effect: 67.39 -> 67.64, the nominal best of the family, though the margin is within run-to-run variability. The on-policy pairs come from the model's competence frontier (instances of variable outcome).

All numbers are clemscore / statscore on the playpen validation split, measured in a single frozen environment (Python 3.11, clemcore pinned via playpen, clembench pinned requirements) with two upstream fixes applied: a division-by-zero guard in the privateshared Game Master and the punkt_tab NLTK resource for the IFEval scorer. Earlier revisions of this card reported numbers from an unpinned environment; see the paper for the environment-sensitivity analysis.

Checkpoint family (LM Playschool challenge, team DAIR)

Base model: Qwen3.5-2B (13.63 / 44.22 in the same environment). Paper: Raising a Small Language Model: From Imitation to Curiosity in Dialogue Games (LM Playschool Challenge 2026). <!-- TODO: add link -->