wAI-org/swerl-tmax-15k-rubric-gpt-5-6-sol

swerl-tmax-15k with task-quality rubric labels (gpt-5-6-sol)

hamishivi/swerl-tmax-15k, unchanged and unfiltered, with a per-task quality label attached as extra columns.

This is not a verified or filtered dataset. Every one of the 14,601 original records is present. Nothing has been dropped, repaired, or reordered. The labels are one model's judgement about whether each task is sound enough to be useful RL training data — an annotation layer, not a correctness guarantee.

Why this exists

The tmax-15k generation pipeline deliberately skips teacher-based correctness validation; rl_data/README.md states that "task quality is left to RL's soft filtering". This dataset supplies the missing quality signal after the fact so that filtering becomes a deliberate downstream choice rather than an implicit one.

What the grader saw

A tmax-15k task is defined by exactly three files: instruction.md (what the agent is told), setup.sh (how the container is built) and tests/test.sh (the verifier that decides reward). The container image contains only fixtures — no verifier — so those three files are the complete task definition. The grader received all three in full, plus task_id, image tag, env_name and source.

instruction.md is verbatim inside the RL prompt for all 14,601 records, so the grader read exactly what the agent reads.

Labels

• —CLEAN — instruction is clear and self-sufficient, and the verifier genuinely validates correctness (recomputes the answer, diffs an oracle, fuzzes) rather than checking shape. A hard-but-fair task is still CLEAN.
• —VERIFIER-TOO-WEAK — the verifier passes without the task being solved: shape/format only, a self-reported value, a keyword grep, a hardcoded subset, or fixed inputs whose answer can be hardcoded.
• —INSTR-VERIFIER-MISMATCH — the verifier checks something the instruction does not imply, so a faithful solution still fails.
• —INSTR-ENV-MISMATCH — instruction or verifier references paths, files or services the environment does not provide.
• —ANSWER-LEAK — the instruction, or a file shipped in the environment, reveals the expected answer.
• —INSTR-AMBIGUOUS — a necessary specification is missing, so two reasonable readings give different outputs and only one passes.
• —TASK-TRIVIAL — no meaningful work or learning signal.
• —OTHER — a genuine defect fitting none of the above.

Columns

Original columns are carried verbatim, so this remains a drop-in for training: messages, ground_truth, dataset, env_config, source.

Added:

The two keep_policy_* columns are conveniences, not recommendations. They encode a judgement about whether instruction fidelity or learning signal matters more, which is a decision for the consumer.

Method and limitations

• —Grader openai/gpt-5-6-sol-genai-responses via MetaGen, one pass per task, no ensemble. Full task definition in context (mean ~2,705 input tokens).
• —Coverage is complete: 14,601 of 14,601 graded.
• —Read `verify_confidence` with care. It is ~0.98–0.99 for essentially every label, so it does not discriminate and should not be used to triage.
• —Single-grader risk is real. On a 20-task calibration, claude-4-6-opus was 100% self-consistent yet demonstrably lenient — it labelled CLEAN a task whose setup.sh ships a working stripped binary that the verifier merely diffs against, so copying it passes with zero work. Self-consistency measures stability, not correctness. gpt-5-6-sol was the most self-consistent of the models tested (18/19) and agreed 18/19 with gpt-5-5, but a single grader can still be systematically wrong.
• —A large share of VERIFIER-TOO-WEAK is one specific shape: the verifier validates the answer but never the artifact or method the instruction demanded. Whether that disqualifies a task depends on your goal. keep_policy_b exists to separate that slice.
• —Generation axes from the source pipeline (domain, verifier_kind, fixture_kind, complexity) are not present in the published tmax-15k, so results cannot be sliced by axis.

Reproducing

Tooling lives in rl_data/verify/ on the verify/tmax15k-rubric branch of the private tmax repo: build_index.py, rubric.py, metagen.py, label_tasks.py, export_hf.py, plot_labels.py.