agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp20-base

seededrlbaseramp25stoppengen4kep2ncp20base — RL checkpoint (global step 24)

Role in this run: best by pass@8

What it is

An OpenRLHF GRPO reinforcement-learning checkpoint for Qwen3-4B.

• —Base model: Qwen/Qwen3-4B-Instruct-2507
• —Seeded from: base Qwen3-4B (no SFT seed — RL applied directly to the base model)
• —Saved at global step 24 of RL run seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp20_base.

What it's best at

• —This is the best checkpoint by pass@8 so far in this run (eval_default_pass8 = 0.2261).

Trained and validated on the cobalt-train ≤2/64 frontier (canonical cleaneval prompts): 1833 train / 112 held-out val problems the base model solved on at most 2 of 64 samples under the iidcanonical@64 hardness scan. Val evals sample at temperature 1.0 (matching the clean_eval frontier eval).

Reward signal: binary code-correctness (1.0 if the generated program passes the problem's tests, otherwise 0.0).

Eval metrics at this checkpoint (held-out val, 8 samples/problem):

• —pass@1 (unbiased; mean per-problem fraction correct): 0.0810
• —pass@8 (problem counted solved if any sample is correct): 0.2261

Where the logs are

• —Weights & Biases: project eaiexp-paper-final, run name seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp20_base.
• —Local train log: experiments/cobalt_qwen3_4b_ft/rl_runs/qwen3_4b_instruct_2507_cobalt_v1/seeded_rl_base_ramp25_stoppen_gen4k_ep2_ncp20_base/openrlhf_train.log

Recipe

• —Algorithm: GRPO (group-normalized advantages, no KL penalty).
• —Stop-properly penalty: truncated samples' reward set to -1.0 (ProRL-style anti-truncation shaping).
• —DAPO overlong penalty: responses in the last 1024 tokens before the cap get an additive penalty ramping to -0.25.
• —Samples per prompt: 8
• —Rollout batch size: 128, train batch size: 128
• —Max new tokens per rollout: 4096
• —Episodes: 2
• —Actor learning rate: 1e-06 (constant schedule)

How to load

This checkpoint is the `main` revision (git branch) of the repo, with the model at the repo root — load it directly, no subfolder:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp20-base", revision="main")
tokenizer = AutoTokenizer.from_pretrained("agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp20-base", revision="main")

Or serve with vLLM: vllm serve agurung/cobalt-seeded-rl-base-ramp25-stoppen-gen4k-ep2-ncp20-base --revision main