Shamima/lm-playschool-qwen3.5-2b-sft-iter2

Qwen3.5-2B SFT for LM Playschool Challenge (iter 2)

A LoRA SFT fine-tune of Qwen/Qwen3.5-2B on the success-filtered split of colab-potsdam/playpen-data, submitted to the LM Playschool Challenge.

Model: Shamima/lm-playschool-qwen3.5-2b-sft-iter2 Code + run history: github.com/silvererudite/lm-playschool-submission

Headline results

Evaluated with playpen eval --suite all. Per-game and per-benchmark breakdowns in `results/iter2.md`.

For context, the published leaderboard baselines for the same eval suite are:

Training methodology

SFT with LoRA, completion-only loss disabled (effectively standard language-modeling loss across the full chat-templated sequence), single epoch over success-filtered episodes. Standard hyperparameters with two non-default choices that were critical to the result (see "Design decisions" below).

Data usage

• —Source: colab-potsdam/playpen-data, interactions split, train portion (34,909 episodes total).
• —Filter: kept only episodes with meta.outcome == "success" → 20,202 episodes.
• —Eval split: none. Iter 1 used a 20% holdout for in-training eval; iter 2 disabled in-training eval (it OOMed on A10G; eval-mode batches don't use gradient checkpointing). All 20,202 episodes used for training.
• —Tokenization: Qwen's chat template applied. Sequences truncated to max_length=1024, which covers ~82% of the dataset uncut (verified empirically — see `scripts/dataset_length_stats.py`; iter 1 used 300, which truncated 64% of episodes — see iter-1 writeup).
• —No external data, no test contamination. Only the public playpen-data train split was used.

Hyperparameters

Compute budget

Reproducibility

The repo at github.com/silvererudite/lm-playschool-submission contains:

• —scripts/setup.sh — clones playpen + clembench, builds venv, installs deps, applies our config patches. Idempotent.
• —scripts/train_sft.py — the full trainer (CONFIG block at top exposes every hyperparameter).
• —scripts/merge_and_push.py — LoRA merge + HF Hub upload.
• —scripts/dataset_length_stats.py — episode-length analysis used to pick max_length.
• —scripts/post_train_pipeline.sh — end-to-end orchestrator (waits on training pid → merge → push → eval).
• —RUNS/iter1.md — iter 1 writeup with the failure analysis that motivated iter 2's hyperparameters.
• —results/iter2.md — full per-game / per-benchmark breakdown of this model.

To reproduce iter 2 exactly:

git clone https://github.com/silvererudite/lm-playschool-submission && cd lm-playschool-submission
bash scripts/setup.sh
# paste your HF token into playpen/key.json
source env.sh
python -m accelerate.commands.launch --num_processes 4 --num_machines 1 \
    --mixed_precision bf16 \
    "$(which playpen)" run scripts/train_sft.py -l Qwen3.5-2B

Design decisions

These are the choices that mattered most (deltas vs the upstream examples/trl/sft_trainer_lora.py defaults):

1. 1.`modules_to_save=[]` instead of upstream's `["lm_head", "embed_token"]`. This was the load-bearing fix between iter 1 (regression to clemscore 3.86) and iter 2 (gain to 46.66). Reasons:
2. 2.Typo bug: the upstream string is embed_token (singular). Qwen's actual module is embed_tokens (plural), so peft silently ignored that entry — only lm_head was made trainable.
3. 3.Tied-weight breakage: Qwen3.5 ships with tie_word_embeddings=True (the input embedding is the output head). Adding lm_head to modules_to_save causes peft to set tie_word_embeddings=False and create a separate trainable copy. After thousands of steps the head drifts away from the still-tied embedding, breaking an architectural invariant the model was pretrained against.
4. 4.With modules_to_save=[], training is pure low-rank LoRA — no full-rank parameters touched, tied weights preserved.

1. 1.`max_length=1024` instead of upstream's `300`. Empirical: median episode length in the success-filtered dataset is 462 tokens, p75 is 814, p90 is 1373. At 300, ~64% of training sequences had their assistant turn truncated; with TRL's effective full-sequence loss, that's ~64% of training compute spent on prefixes that never reached the target. Bumping to 1024 covers ~82% of episodes uncut. 2048 would cover ~95% but costs ~4× attention memory; we left the headroom for iter 3.

1. 1.`eval_strategy="no"` (no in-training eval). TRL's eval loop disables gradient checkpointing for eval batches. On 4× A10G this OOMed at the first epoch boundary (iter 2's first attempt crashed at step 2021/6063 trying to allocate +7.58 GB on top of 15.85 GB already in use). We measure model quality via the full clembench eval after training, so eval_loss during training adds nothing.

1. 1.`save_strategy="steps"`, `save_steps=1000`, `save_total_limit=2`. Crash-recovery insurance. With save_strategy="epoch" the first checkpoint wouldn't appear until ~1.5 h in; on shared infrastructure that's a long uninsured stretch.

1. 1.DDP via accelerate with a small monkeypatch in train_sft.py: clemcore's huggingface backend hardcodes device_map="auto", which pipeline-shards the model across all visible GPUs and runs only one GPU at a time. Under accelerate (LOCAL_RANK set), we override AutoModelForCausalLM.from_pretrained to use device_map={"": LOCAL_RANK} so each rank holds a full model copy on its own GPU. Single-process pipeline-parallel iter 1: 4 h. 4-GPU DDP iter 2: 2 h 36 m.

1. 1.Generation-config fix on the merged model. Qwen3.5-2B's text_config.eos_token_id is 248044 (<|endoftext|>), but its chat template terminates with <|im_end|> (248046). After SFT, the model emitted <|im_end|> correctly but model.generate() didn't stop there (wrong eostokenid), causing the model to "continue" past its turn and hallucinate fake user/assistant exchanges. Fixed by setting eos_token_id: [248046, 248044] and pad_token_id: 248044 in generation_config.json of the uploaded model. This is a property of the model, not an eval-pipeline trick.

Limitations and known regressions

• —referencegame quality 100 → 8.33: largest per-game regression. SFT learned a verbose explanation style from playpen-data that hurts on the single-turn exact-match protocol of referencegame.
• —ifeval 65.22 → 36.96, eqbench 65.01 → 46.77: SFT softened the model's instruction-following discipline. The base model's strict format adherence on these benches partially eroded.
• —wordle unchanged at 33.33 % played, 0 quality: the model now follows wordle's protocol but doesn't have the world-model to actually win wordle.

License

Apache 2.0 (inherited from Qwen3.5-2B and from the LoRA-only adapter on top).