CoolFace
Modelpublic

andrewlngdn/dsl-debug-7b-rl-only-step30

sourceHugging Facemitupdated 6mo agoView on Hugging Face
0likes16downloads
Model Card

DSL Debug 7B — RL-Only Step 30

Qwen2.5-7B-Instruct trained with GRPO (Group Relative Policy Optimization) directly from the base model, no SFT warmup.

Blog post: Multi-Turn RL for Code Debugging Code + environment: github.com/AndrewLngdn/dsl-debug

Training

  • —Method: GRPO with multi-turn tool use (verl 0.7 + sglang 0.5.6)
  • —Base model: Qwen2.5-7B-Instruct (no SFT warmup)
  • —Steps: 30 (batch size 512, 8 rollouts per prompt)
  • —LR: 1e-5 cosine
  • —Reward: Binary (1.0 if submitted code matches expected output, 0.0 otherwise)
  • —Hardware: 2x A100-SXM4-80GB

Results (held-out test, one-shot)

SplitBase ModelThis Model
Standard (481)50.5%78.8%
Nonlocal (200)12.0%54.0%
Intent-Mismatch (177)0.6%14.7%

Alignment Tax

BenchmarkBaseThis Model
MMLU (5-shot)74.6%74.7%
GSM8K (8-shot)84.9%84.4%
HumanEval (0-shot)65.9%59.1%

Usage

python
from huggingface_hub import snapshot_download
snapshot_download("andrewlngdn/dsl-debug-7b-rl-only-step30",
    local_dir="/workspace/models/rl_only_step30")

Related Models

ModelRepo
SFT then RL step 35 (best)andrewlngdn/dsl-debug-7b-sft-rl
SFT step 100andrewlngdn/dsl-debug-7b-sft-step100