wAI-org/swerl-tmax-15k-solvable-gpt-5-6-terra

swerl-tmax-15k hardened, post-validation-filter (dataset 3 of 3)

Which tasks in hamishivi/swerl-tmax-15k can a strong model actually solve? Every task was attempted twice as a full agentic episode — real sandbox, real bash, real verifier — and a task is verified when at least one attempt earned reward.

The last of three artifacts that exist to be compared by task_id:

1. 1.original — `hamishivi/swerl-tmax-15k`, unchanged — 14,601 tasks
2. 2.hardened, pre-validation-filter — `wAI-org/swerl-tmax-15k-hardened-prefilter` — 13,565 tasks (+1,036 in its exclusions config)
3. 3.hardened, post-validation-filter — this dataset, a strict subset of 2 — 7,015 tasks

Supporting artifacts: rubric audit · proposed repairs

This filter selects tasks with a demonstrated passing solution under this model and budget (two attempts, 600 s wall-clock, 40 steps). Two failed attempts do not establish that a task is unsolvable.

Rollouts ran against the repaired tasks where a repair existed: hardened verifier, and for answer leaks a rebuilt image with the leak removed.

A zero is not a zero

The single design decision that matters here. "The model failed" and "we never actually got a measurement" are different claims, and collapsing them silently deletes good tasks:

not_evaluable is not a rounding error. The grading route content-filters security-flavoured tasks: 95% of its hard failures contained terms like exploit, vulnerability, bypass, or privilege, against a 28% corpus baseline. Scoring those as failures would have quietly stripped the security subset out of the dataset. They are labelled and left alone; a second model (gemini-3-8-flash-genai) recovered some, and its verdicts are included.

The 600s cap is safe by measurement, not assumption: of 12,748 solved episodes, exactly one exceeded 600s end to end (at 659s) — a 0.01% cost.

Configs

• —`default` — the filtered subset: tasks with a demonstrated passing solution.
• —`evidence` — every task that was attempted, with its outcome tally, so the filter can be audited rather than taken on trust.

Results

51.8% of the 13,535 tasks with real evidence have a demonstrated passing solution, from 28,266 scoring episodes (29,866 including the A/B control arm). The second model mattered: the fallback pass cut not_evaluable from 92 tasks to 24, recovering evidence on tasks the primary route would not touch.

Solve rate vs. the independent rubric audit

Two filters built from different signals agree in direction: tasks the rubric called CLEAN solve at 79%, against 47% for tasks whose verifiers were hardened. 5,170 tasks were solved through a hardened verifier, so those verifiers are demonstrably satisfiable — which the repairs dataset flagged as unproven.

But the hardening is not free — a controlled A/B

The raw gap above is confounded with task difficulty, so it was measured directly: 800 hardened tasks re-run with their original verifier. Same tasks, same images, same model, same budget; only the verifier differs.

On 790 paired tasks: 369 pass both, 237 pass neither, 166 flipped pass→fail, 18 flipped the other way (a ~2.3% noise floor).

The comfortable explanation is that those 166 were loopholes. The evidence does not support that. If the original verifier had been rubber-stamping, the flipped tasks should have been solvable in very few steps. They were not — they took more work than the tasks that still pass:

Their traces show the agent writing real Rust, C and C++ before submitting. Some of that 166 is genuine loophole-closing on harder tasks; some is likely over-hardening, and this data cannot separate the two. Treat hardened-verifier tasks marked unverified as needing review, not as confirmed-bad — and prefer CLEAN tasks where a clean signal matters.

Columns

Caveats

• —Solvable is not the same as well-posed. A verified task was solved once; it may still be ambiguous, or its verifier may still be loose. This filter removes the unsolvable, which is a floor on quality, not a ceiling.
• —Two attempts is a low bar. A task marked unverified may well be solvable with more samples or a different model. Read it as "not demonstrated", not "impossible".
• —no_submit (the agent never submitted within 40 steps) counts as an attempt. It is weak negative evidence and is tallied separately so it can be excluded.

Runtime fixes applied before this filter

A bounded runtime pass executed every rewritten verifier against an untouched sandbox. 899 of 12,275 were defective — mostly a pre-existing undeclared dependency (662 verifiers import requests, absent from the image; the original verifiers import it too, 662/662, so those tasks could never pass in the original corpus either). 710 were fixed by baking the dependency into the image at build time, and every changed task was re-validated on its final version before being admitted here.

Reproducing

rl_data/verify/ on branch verify/tmax15k-rubric of the private tmax repo: rollout.py (harness), materialise.py (repaired corpus), export_verified.py. Episodes ran on the apptainer backend inside the job's own Slurm allocation — no shared sandbox daemon — so the pass could not disturb concurrent RL training.

Before you train on this

Two properties of tasks.jsonl that are not obvious from reading a row:

• —`env_config` is a dict, matching `hamishivi/swerl-tmax-15k`. It was a JSON string until 2026-09-15. On revisions before that, grpo_fast's _normalize_env_config_column raises TypeError: env_config must be a dict, list, or None, got str — on every row, so the job dies in the first minutes. Use the current revision, or coerce the column yourself.
• —`env_config.image` is an absolute path on the cluster this was built on (/checkpoint/memorization/oscaryinn/...) and will not resolve anywhere else. Retarget it before use. `wAI-org/tmax-image-pool` ships task_to_blob.json mapping every task id to its blob.

The field is a path rather than a registry reference on purpose: a large share of these tasks use derived images that exist in no registry, and prefer_local_sif() passes an absolute path straight through with no lookup. There is no single value that is correct on every machine, so retargeting is a required step rather than an oversight.