sergiopaniego/qwen3-1.7b-mbpp-grpo

qwen3-1.7b-mbpp-grpo

Qwen3-1.7B trained with GRPO inside an OpenEnv coding environment, using TRL. Built for Class 4 of the Training Agents series.

The model solves MBPP problems inside a live Python session. A run_python tool executes code in that session and state persists across calls, so the model can define a function, test it, read the error and fix it. The environment owns the reward: it runs the task's hidden tests in the same session and returns the fraction that pass, which is why the training script has no reward_funcs at all.

Results

Fraction of hidden tests passed on the 257 problems of MBPP sanitized/test, which never appear in training. Both models were scored in the same pass, so the difference is not comparing across eval runs:

The same run also scored a fixed set of 64 held-out problems every 5 steps during training, which is the curve that actually shows the learning:

The training reward looks flat because each GRPO step here scores only 4 problems, and MBPP problem difficulty varies far more than the model improves over one run. Step-to-step noise is 0.213 against an effect of about +0.06, so that curve has no power to resolve it. Watch eval/reward.

Dashboard: trackio-training-agents-4, run mbpp-grpo. The weights, the curve and the score above all come from that one run.

Reproducing it

hf jobs uv run --flavor a100-large --timeout 90m -s HF_TOKEN \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 --with "trl[vllm]" \
  "https://huggingface.co/sergiopaniego/rl-envs-youtube-livestream-4-scripts/resolve/main/train_coding_whitebox_eval.py" \
  -- --model Qwen/Qwen3-1.7B \
     --env-url https://sergiopaniego-coding-env.hf.space \
     --trackio-space-id sergiopaniego/trackio-training-agents-4 \
     --push-to-hub --hub-model-id <your-username>/qwen3-1.7b-mbpp-grpo

170 train problems (MBPP sanitized train + validation + prompt, none of them in the eval split), 42 steps, 4 problems per step, 4 generations each, lr 1e-6, about 60 minutes on one A100.

• —Scripts, with the eval script and the sizing gotchas: rl-envs-youtube-livestream-4-scripts
• —Environment: coding-env

What to know before you read the numbers

• —The dataset has a ceiling of 0.918, not 1.0. Running MBPP's own reference solutions inside this environment's sandbox, 14 of the 170 training problems are impossible: 11 need imports outside the executor's allowlist, three hit interpreter limitations.
• —About 32% of the reward is a test the model was shown. The "Example test" in the prompt is test_list[0], and that same test is one of the roughly 3.2 tests that get scored, in all 170 problems.
• —The size of the improvement moves between runs. Two runs of this exact config gave held-out rises of +0.095 and +0.118, both at t ~ 5.2, and 257-problem deltas of +0.064 and +0.100. The rise reproduces, its magnitude moves by about a third. Do not read one run's number as the number.

Usage

from transformers import pipeline

generator = pipeline("text-generation", model="sergiopaniego/qwen3-1.7b-mbpp-grpo", device="cuda")
print(generator("Write a python function to find the first repeated character in a string.")[0]["generated_text"])

The model was trained to work through a tool in a live session, so it is at its best driven the way it was trained, with run_python available and the hidden tests scoring the session afterwards. The snippet above is a smoke test, not the intended use.

Citations

GRPO, from DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.