Datasetpublic

moofeez/llm-debugger-eval-transcripts

llm-debugger evaluation transcripts Every turn behind the results reported in llm-debugger: the base model, the SFT initialisation, and the RL policies trained from it. Exploratory runs no reported figure depends on are not included. Layout path what runs/base/ Qwen3-Coder-30B-A3B-Instruct, 8 runs on the 30-task test split runs/sft/ the SFT initialisation, 3 runs on the test split runs/rl-gate-arc/ the RL gate arc, v15 through v120 (3 runs each, 8… See the full description on the dataset page: https://huggingface.co/datasets/moofeez/llm-debugger-eval-transcripts.

sourceHugging Faceapache-2.0updated 15d agoView on Hugging Face

0likes83downloads

Dataset Card

llm-debugger evaluation transcripts

Every turn behind the results reported in `llm-debugger`: the base model, the SFT initialisation, and the RL policies trained from it. Exploratory runs no reported figure depends on are not included.

Layout

path	what
`runs/base/`	`Qwen3-Coder-30B-A3B-Instruct`, 8 runs on the 30-task test split
`runs/sft/`	the SFT initialisation, 3 runs on the test split
`runs/rl-gate-arc/`	the RL gate arc, v15 through v120 (3 runs each, 8 at v90)
`runs/rl-v90-test/`	RL policy v90, 8 runs on test
`runs/rl-v90-val/`	RL policy v90, 8 runs on the 40-task validation split
`runs/sft-trajectories/`	the SFT model's runs turn by turn

runs/sft/ is the gate at policy v0, which carries a zero-initialised RFT delta, so it measures the frozen SFT model exactly.

One archive per run. manifest.json lists file counts and uncompressed sizes. Fields naming the run directory, such as run_dir, are rewritten to the published name so they resolve against what you extract.

bash

tar -xzf runs/rl-v90-val/v90_val_a_run1.tar.gz

Inside a run: combined_results.json (full conversation_log per episode), <model>.json (per-defect records), evaluation_summary.json, per_task_summary.json.

Reading these

Solve rate is final_test_passed, not anything the model claims. Trajectory files carry success, which is not the solve metric — it requires the model to have called done, so an episode that hit the turn cap with the suite passing reads success: false and final_test_passed: true. One of the 90 SFT trajectories here is that case. Score from the run records; read the trajectories for behaviour.

The gate arc is included as evidence for a caveat, not a result. v90 was chosen by stop-at-peak on that arc, and the arc ran on the pristine test split:

v0 64.4 | v15 65.6 | v30 63.3 | v45 70.0 | v60 62.2 | v75 72.2 | v90 76.7 | v105 68.9 | v120 66.7

So v90's test figure is the argmax of nine noisy draws on the set it was then scored against. Cite the validation number, 75.9 / 93.1.

Use

python

from huggingface_hub import snapshot_download
path = snapshot_download("moofeez/llm-debugger-eval-transcripts", repo_type="dataset")