Nyandwi/qwen3.5-2b-data-agent-subset100-eval

Qwen3.5-2B on data_agent_rl_environment_train_subset_100

pass@1: 64/100 = 64% · model Qwen/Qwen3.5-2B · temperature 0.7 · 1 rollout/task · max 10 code turns

Task environments ran as Modal sandboxes (one container per task, Kaggle slice pulled from the HF bucket into /home/user/input). Rewards come from each task's own tests/grader.py with the LLM-judge tier disabled, so grading is deterministic: exact string match, else numeric match within 1e-3.

Where the score comes from

Failure breakdown

Did the model commit to an answer on its own?

The agent scaffold adds a fallback: if the model burns all its code turns without submitting, it gets one final turn that asks only for the answer. This matters for reading the raw number — a 2B model often computes the right value and then keeps exploring instead of committing.

So 63/100 (63%) were solved with no scaffold intervention; the forced turn recovered a further 1.

Sandbox behaviour

• —Mean code cells executed per task: 2.4
• —Tasks where at least one cell raised: 40/100
• —Tasks that re-ran an identical cell (loop symptom): 4/100
• —Mean wall time per task: 9s end-to-end, including container start and the data pull (the first task per image pays ~30s; Modal's cache amortises the rest)
• —Harness errors: 0

What's actually going wrong

1. Answering before looking. The median rollout runs just 2 code cells. 41 tasks were answered after a single cell, and 13 of those were wrong. Failed rollouts average 3.1 cells vs 2.0 for solved ones — the model either gets it immediately or flails. Typical shape: it picks a plausible-looking column without checking dtypes/head() and aggregates the wrong thing (e.g. reporting the max of the wrong price column, or counting rows after a filter the question never asked for).

2. Precision conventions, not reasoning. 6 of the 8 close_miss failures are the model being more precise than the gold: the gold answers were rounded by the original notebook author, and the grader's 1e-3 tolerance rejects the full-precision value. These are arguably dataset artifacts rather than model errors — worth deciding explicitly before using this reward signal for RL, since it teaches the model to guess a rounding convention.

3. Bailing out. 3 tasks were answered Not Applicable on questions that were perfectly answerable — in one case the answer was simply the row count. A further 1 returned an uninterpolated template placeholder instead of a value.

4. Tool use is mostly fine. 40/100 tasks hit at least one Python exception, and failed rollouts raise more often (0.86 vs 0.56 exceptions), but exceptions are a symptom of flailing rather than the primary cause. Only 4 tasks showed the repeat-the-same-cell loop.

All 100 tasks

R = reward. Answers and questions are truncated for readability; full trajectories are in trajectories/.