Modelpublic

VityaVitalich/1pp-1.7b-asst-sft-grpo-gsm8k

sourceHugging Faceupdated 13d agoView on Hugging Face

0likes129downloads

Model Card

1PP 1.7B (assistant-conditioned) — GRPO on GSM8K

GRPO fine-tune of `Raghav-Singhal/1pp-1.7b-asst-sft` on GSM8K, 100 policy steps. Trained with verl 0.9.0.dev0 on 4x GH200.

One of three siblings that differ only in the pre-training persona condition of the starting checkpoint — asst, ua, raw — trained with an identical recipe so the conditions can be compared after RL.

Results

GSM8K test, 16 samples per question, temperature 1.0.

	pass@1	pass@16
before RL (step 0)	0.0110	0.0979
after RL (step 100)	0.0393	0.2008

Full curve:

step	pass@1	pass@16	entropy	resp. chars
10	0.0203	0.1573	1.281	399
20	0.0225	0.1705	1.123	336
30	0.0315	0.2268	0.769	287
40	0.0286	0.1722	0.559	249
50	0.0376	0.2155	0.451	231
60	0.0332	0.2097	0.388	225
70	0.0312	0.1958	0.358	213
80	0.0422	0.2110	0.339	206
90	0.0352	0.1957	0.319	212
100	0.0393	0.2008	0.293	223

Caveat on comparing the three siblings

A configuration-identical repeat of the asst arm reached pass@1 0.0552 at step 100 where this run reached 0.0393. That seed-to-seed gap is larger than the spread across the three conditions (0.0369-0.0413), so the ranking between asst, ua and raw here is not evidence of a real difference. Treat each model on its own absolute improvement, not against its siblings, until multiple seeds exist.

Checkpoints

main is step 100. Every earlier checkpoint is a branch:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "VityaVitalich/1pp-1.7b-asst-sft-grpo-gsm8k"
model = AutoModelForCausalLM.from_pretrained(repo)                      # step 100
model = AutoModelForCausalLM.from_pretrained(repo, revision="step-50")  # step 50
tok   = AutoTokenizer.from_pretrained(repo)

Branches: step-10, step-20, step-30, step-40, step-50, step-60, step-70, step-80, step-90, and main (= step 100).

Training


algorithm	GRPO (group-relative advantages, no critic), KL penalty retained
group size (`rollout.n`)	16
prompts per step	256
mini-batch	128 prompts
`ppo_epochs`	3
learning rate	1e-6, constant
max prompt / response	256 / 512 tokens
reward	verl's stock GSM8K scorer, unmodified
precision	bf16
hardware	4x GH200, ~50 s/step

ppo_epochs=3 was chosen by sweep: 1, 2, 4 and 5 all validate worse at matched steps, and 4 and 5 reach a higher training reward while validating lower — the signature of over-reusing a fixed batch of rollouts.

Notes

Weights are bf16. verl exports FSDP's fp32 master copy and declares bfloat16 in config.json; the cast here makes the file agree with the config and is lossless for inference. The tokenizer files come from the base repo unchanged — RL does not touch the tokenizer, and verl's export omits four of them.