abhid1234/flywheel-rl-benchmark

flywheel — a benchmark for RL on agent trajectories

A small, controlled benchmark for studying whether a coding agent can improve from its own graded failures — reinforcement learning at the context layer (the policy update is a durable lesson carried in context, not a weight change) — together with the baseline results from running it live on Daytona sandboxes with a real coding agent.

Companion to [github.com/abhid1234/flywheel](https://github.com/abhid1234/flywheel). The full method and honest write-up: FINDINGS.

Why these tasks are interesting

A strong model aces ordinary code-gen tasks at baseline — there is nothing to learn. Real learning only shows up at the model's failure frontier. The conventions split puts it there: each spec is deliberately underspecified on one point, and a hidden test enforces a convention the model cannot infer (empty input → None, "unique" → sorted, ranges inclusive of both ends, 1-indexed positions, case-insensitive matching, ISO dates, …). Cold, a competent model guesses — and often guesses wrong. Once it has learned the convention, it complies. This is how an agent meets a codebase's implicit rules: by getting them wrong first, then learning from the failure.

Splits

Each row:

{
  "id": "nth-item",
  "mode": "one-indexed",
  "split": "train",
  "spec": "solve(items, n) → the item at position n.",
  "hidden_test": "from solution import solve\nassert solve(['a','b','c'],1)=='a'\n...",
  "reference_broken": "def solve(i,n):\n    return i[n]",
  "reference_fixed": "def solve(i,n):\n    return i[n-1]"
}

• —`spec` — the (underspecified) prompt given to the agent.
• —`hidden_test` — the grader. Kept out of the agent's reach; this is the verifiable reward.
• —`reference_broken` / `reference_fixed` — a buggy and a correct solution, for validating a grader offline with zero LLM cost (broken must fail, fixed must pass).
• —`mode` — the recurring failure a single durable lesson repairs; `split` — train / holdout. Improvement is only ever measured on the sealed holdout split the agent never learns from.

Results (results/)

Baseline runs with a live Codex agent, code executed against the hidden tests in isolated Daytona sandboxes. Reported honestly, including the parts that refuted the hypothesis:

• —`live-curve.json` — one live run: 27% → 76% on the sealed held-out set, learning lessons the model wrote itself.
• —`replication.json` — 4 independent runs: every run climbed (+27 to +48pp) from an identical baseline. The mechanism reproduces; which individual lesson lands varies with run-to-run noise.
• —`sweep-k04..k16.json` — a rollout sweep. The final does not rise with compute (73/67/66/59%); what scales, exactly as 1/√K, is measurement precision (the noise band: 12.7 → 8.9 → 7.3 → 6.3pp). More compute buys a sharper ruler, not a faster runner.
• —`credit-real.json` — snapshot/replay credit assignment localizing which step of a failed trajectory caused it.

The one rule

The model writes the fix. It never writes the success criterion.

The reward comes from the hidden tests, fixed in advance and out of the model's reach. A lesson is only credited when its gain on the sealed held-out set clears a measured noise floor — the loop refuses to credit an improvement it cannot distinguish from noise.

Citation

flywheel: agents that improve from their own production traces.
https://github.com/abhid1234/flywheel  ·  MIT.