OctoReasoner/effibench_verl

EffiBench (verl efficiency eval set)

A canonical-validated subset of DONG19/EffiBench (arXiv 2402.02037; 1000 LeetCode-style Python tasks), converted to the verl rule-reward schema by verl/scripts/data/effibench.py. License: unspecified upstream (no license tag on the HF dataset or the GitHub repo) -- treat as research-only until clarified.

Row filter (854/1000 kept). The upstream test_case column is broken for a sizeable minority of rows: tree/linked-list tasks contain literal <__main__.TreeNode object at 0x...> reprs inside asserts, in-place tasks assert on never-defined variables, 14 rows have no tests at all. The filter is executable, not heuristic: a row is kept iff its own canonical solution passes its own assert suite through the actual grading harness (974 structurally usable, 854 pass). The canonical's build-machine runtime/peak-RSS are stored in extra_info.reference_runtime_ms/reference_memory_mb as fallback references.

Scoring (verl/verl/utils/reward_score/effibench.py): the extracted class Solution runs the assert suite in a firejail execution and is compared against the canonical solution re-measured on the same machine (cached per task): reward = 0.5*pass + 0.1*min(1, canon_rt/rt) + 0.4*min(1, canon_mem/mem). Eval recipe: verl/recipe/run_effibench_eval.sh.

Efficiency is the whole point of EffiBench, so it is measured on the axis that is actually reliable. Memory (weight 0.4) is tracemalloc peak -- Python-level allocations, deterministic across runs (measured: 0% variance) and free of the ~20MB interpreter baseline, so it cleanly separates e.g. O(n) from O(n^2) space. This is sharper than the paper's own memory_profiler TMU/NMU. Runtime (weight 0.1) is the min of 5 timed runs, kept as a light secondary because EffiBench's inputs are sub-millisecond and wall-clock runtime has 5-10% run-to-run noise -- which is why the official per-task runtime has ~0 test-retest reliability. That noise is a property of the dataset's tiny inputs, not the harness, and no measurement method removes it; only aggregate runtime means anything.

Known compromises vs the official pipeline

• —Runtime efficiency is low-resolution (dataset has sub-ms inputs); de-weighted to 0.1 and reliable only in aggregate. Memory (tracemalloc) is the trustworthy efficiency signal.
• —Assert suites are binary (first failing assert aborts): no per-case fraction.
• —Efficiency ratios are clipped at 1 -- beating the canonical earns full credit, not extra credit.
• —Prompts use a class Solution starter skeleton reconstructed from the canonical solution (the source ships no starter-code column), wrapped in the FinalMix3 training format (unified system turn + [INSTRUCTION] block + reminder, verl/scripts/data/unified_prompt.py). Numbers recorded against the earlier house-LCB-style prompts are not comparable.