Modelpublic

while-ai/paper-zero-rl-format-reward-4b

sourceHugging Faceapache-2.0updated 2d agoView on Hugging Face

0likes42downloads

Model Card

paper-zero-rl-format-reward-4b

Recipe: [recipes/papers/zero-rl-format-reward](https://github.com/whilehq/whileai-sdk/tree/main/recipes/papers/zero-rl-format-reward) · Collection: [Papers, replicated](https://huggingface.co/collections/while-ai/papers-replicated-6ab271de22542eb550d4251c)

Zero RL on a base model: GRPO from Qwen3.5-4B-Base on MATH levels 3 to 5, strict boxed reward against correctness only. A rigid format reward costs accuracy. Both arms are read leniently at eval so the target is one number.

Result

Run 2026-09-18, both arms, one H100, 46.2 GPU minutes.

Arm	pass@1	95% CI	pass@4	Steps	GPU min
Base, no training	0.51	[0.45, 0.57]	0.74	0	0
Baseline (strict boxed reward, -1 without a box)	0.63	[0.57, 0.68]	0.80	30	26.4
Recipe (correctness only)	0.72	[0.66, 0.77]	0.88	30	19.8

Recipe vs baseline: +0.094 [+0.052, +0.139] over 160 paired tasks. Both arms beat the base; the gap between them is the paper's finding and it clears the noise band the three base re-runs set. On the baseline's own strict reward the recipe arm scores lower (0.53 vs 0.58): it boxes less and is right more.

The six seed folders are the 2026-09-21 follow-up the README's Learned section asks for: three training seeds per arm at the same settings. Their paired read-out is not in the recipe README yet; the adapters are here so the numbers can be recomputed.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
`.`	recipe arm: correctness-only reward, 2026-09-18 run (the README's table)
`baseline`	baseline arm: strict boxed reward, 2026-09-18 run
`recipe-seed17`	recipe arm, seed 17, 2026-09-21
`recipe-seed18`	recipe arm, seed 18, 2026-09-21
`recipe-seed19`	recipe arm, seed 19, 2026-09-21
`baseline-seed17`	baseline arm, seed 17, 2026-09-21
`baseline-seed18`	baseline arm, seed 18, 2026-09-21
`baseline-seed19`	baseline arm, seed 19, 2026-09-21

Load

python

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B-Base")
model = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b")  # the headline arm
model = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b", subfolder="baseline")  # another arm

Reproduce

bash

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/zero-rl-format-reward
python recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.