dmaheshwar22/verifiable-rl-coder

verifiable-rl-coder

Live, side-by-side comparison of a base coding LLM, an SFT fine-tune, and a GRPO-trained model — with the full sandboxed test-execution pipeline running in the browser.

This Space is the interactive front-end to a complete open implementation of the verifiable-reward RL post-training technique behind DeepSeek-R1, the OpenAI o-series, and Kimi-K1.5 — applied to a small open coding model (Qwen-2.5-Coder-1.5B). Everything is open: weights, training code, evaluation harness, and the multi-week debugging log of what actually broke and how it got fixed.

Try it

1. 1.Pick Compare (side-by-side) in the sidebar.
2. 2.Choose Base + SFT (and GRPO once that's available).
3. 3.Use a pre-filled example or write your own coding task + assert tests.
4. 4.Click Generate + run tests — watch each model produce a solution and run it against your tests in a sandboxed Python subprocess.

A few prompts that cleanly differentiate the models:

• —Roman numeral conversion — base often forgets the subtractive-notation pairs (IV, IX, XL, XC, CD, CM); SFT learned them
• —Closest-to-zero with tie-breaking — both fail, but in qualitatively different ways (base writes structurally invalid code; SFT writes the correct algorithm with one inverted comparison)
• —Array rotation with k > len — both miss the modulo; this is exactly the kind of edge-case GRPO is designed to catch via test-execution feedback

Full prompt gallery + reproducible recipes: DEMO_EXAMPLES.md.

What this demonstrates

Technical approach

                ┌──────────────────────────────────┐
                │      Streamlit UI (this Space)   │
                │  prompt + tests → patches + runs │
                └──────────────┬───────────────────┘
                               │
                ┌──────────────▼───────────────────┐
                │           Proposer               │
                │  Qwen-1.5B / +LoRA-SFT / +GRPO   │
                └──────────────┬───────────────────┘
                               │
                ┌──────────────▼───────────────────┐
                │           Verifier               │
                │  subprocess pytest in sandbox    │
                │  (5s timeout, isolated workdir)  │
                └──────────────┬───────────────────┘
                               │ pass/fail + composite reward
                               ▼
   Offline (training, not in this Space):
   GRPO rollout buffer → reward → group-relative advantage → LoRA update

Composite reward — R = 1.0·correctness + 0.05·lint + 0.05·runtime + 0.01·length. Correctness uses real test execution (binary pass/fail in [0, 1]). The 20× weight ratio between correctness and each auxiliary signal mechanically prevents reward hacking via short-but-wrong code or lint-clean stubs. Full breakdown: REWARD_DESIGN.md.

Training data — 319 MBPP-train prompts contamination-filtered against MBPP+ test set, 2,580 rejection-sampled solutions kept after sandboxed test execution.

KL configuration — DeepSeek-R1 style (KL added to loss, not reward), kl_loss_coef = 0.04, kl_loss_type = low_var_kl. Tighter than R1's 0.001 default as a defensive choice given the small training set.

Results (current snapshot)

Evaluated with evalplus at temperature 0.2, n=5 samples per task.

The SFT delta is statistically modest (~3.8 pt noise floor for n=164) — documented honestly in the model card. The qualitative analysis in DEMO_EXAMPLES.md shows where the gain comes from: targeted improvements on problems requiring specific structured-knowledge patterns (Roman numeral subtractive notation, edge-case-aware list operations), with non-destructive behavior on the ~70% of problems where base was already correct.

Open artifacts

• —SFT model: dmaheshwar22/qwen-1.5b-coder-sft-v1 (LoRA adapter, 17 MB, full model card with training data + hyperparameters)
• —GRPO model: dmaheshwar22/qwen-1.5b-coder-grpo-v1 (will go live once 200-step training completes)
• —Source code: github.com/Devesh-Maheshwari/verifiable-rl-coder
• —Training infrastructure: HTCondor submit scripts, verl + vLLM training config, sandbox runner with seccomp / resource limits — all in the repo
• —Debugging log: grpo-chtc-debugging-log.md documents the 12 distinct failure modes hit while getting verl + CHTC + vLLM
• —Hydra to actually run a training step. Most ML projects hide this; we publish it because it's the most useful artifact for someone trying to reproduce the work.

Why ZeroGPU would meaningfully improve this Space

Running on CPU basic, generating a single 512-token response takes 30–60 seconds. The side-by-side compare mode triggers two such generations sequentially — so a recruiter or researcher exploring the demo waits ~90 seconds per click. That latency throws away the demo's actual value: you can't feel the model differences when each comparison takes minutes.

ZeroGPU would change this from "leave a tab open and check back" to "interactive exploration." A T4 / A10 with vLLM does 1.5B inference at ~50 tokens/sec — generations land in 2–4 seconds. The user can run the full DEMO_EXAMPLES.md gallery in 5 minutes instead of 45.

This particularly matters for the comparison-driven nature of this work. The whole pitch is "see how SFT and GRPO change behavior on the same prompt" — that observation is qualitative and requires multiple side-by-side runs. Slow inference makes it impractical at any scale.

Limitations (honest)

• —1.5B parameters — competent on isolated functions, weak on multi-file repositories or large-context reasoning. Don't expect SWE-bench wins.
• —319-prompt training set — small; gains are bounded; we surface this explicitly in REWARD_DESIGN.md rather than oversell.
• —MBPP-shape distribution — model is best on problems matching its training distribution (algorithmic Python functions with assert tests). Less reliable for systems code, async, or competitive-programming-heavy problems.
• —Inherits Qwen-2.5-Coder base properties — including any biases or safety properties of the upstream model.
• —CPU inference is slow — see "Why ZeroGPU" above.

Educational value

This Space + the connected GitHub repo + the model card together form a complete reference implementation of small-scale verifiable-reward RL post-training. Specifically useful for:

• —Researchers / students who want to read the full pipeline end-to-end without paywalls or proprietary internals
• —Engineers studying how reward hacking is prevented mechanically (weight ratios in composite reward, KL configuration, length monitoring)
• —Anyone investigating why small fine-tunes plateau and what GRPO is designed to fix beyond imitation learning

The connected docs (REWARDDESIGN.md, DEMOEXAMPLES.md, the debugging log, ablation tables in EVAL_RESULTS.md) are written specifically to be readable without prior frontier-RL context. The composite reward formula, the DeepSeek-R1 KL configuration choice, the sampling-temperature observation on base/SFT comparison — all explained from first principles.

Acknowledgements

Built on:

• —Qwen-2.5-Coder-1.5B-Instruct — base model
• —verl — production GRPO trainer
• —vLLM — fast rollout engine
• —TRL + PEFT — SFT + LoRA
• —evalplus — robust HumanEval+/MBPP+ evaluation
• —DeepSeek-R1 — methodology reference

Trained on the UW-Madison CHTC cluster.

Citation

@misc{maheshwari2026verifiable,
  title  = {verifiable-rl-coder: GRPO post-training of small coding LLMs with sandboxed test-execution rewards},
  author = {Maheshwari, Devesh},
  year   = {2026},
  url    = {https://github.com/Devesh-Maheshwari/verifiable-rl-coder}
}