OctoReasoner/mercury_verl

Mercury (verl efficiency eval set)

The eval split of Elfsong/Mercury (arXiv 2402.07844; 256 LeetCode-style tasks; the train split ships no test cases and is not gradable), converted to the verl rule-reward schema by verl/scripts/data/mercury.py. Source license CC-BY-NC-4.0 (non-commercial) -- this conversion keeps that license.

Every row's ground truth carries the full official scoring contract: entry point, the task's convert_offline/evaluate_offline hooks (lctk linked-list / binary-tree conversions included), the materialised test cases, and the reference solution pool. All 256 rows verified at build time: at least one reference solution passes the grading harness locally.

Prompt format: the FinalMix3 training format -- unified system turn + leading [INSTRUCTION] block + ### Starter Code: section + trailing format reminder (verl/scripts/data/unified_prompt.py) -- so evaluation prompts match the training distribution. Numbers recorded against the earlier house-LCB-style prompts are not comparable.

Scoring (verl/verl/utils/reward_score/mercury.py): the extracted class Solution runs all test cases through the task's own hooks in ONE timed firejail execution, reproducing the official sandbox (github.com/Elfsong/Mercury src/sandbox.py). On a full pass, the official Beyond percentile is computed against the task's reference solutions re-executed on the same machine (cached per task); reward = 0.5*pass + 0.5*beyond. Eval recipe: verl/recipe/run_mercury_eval.sh.

Known compromises vs the official evaluator

• —At most MERCURY_MAX_REFS (default 8) of the ~18 reference solutions are measured per task, sampled evenly across the LeetCode-runtime ordering, to bound reward-time cost.
• —Beyond = clip((max_rt - rt)/(max_rt - min_rt), 0, 1) -- the paper's formula. The reference implementation clamps the numerator at 1 second before dividing (treated here as a bug).
• —Runtimes come from a single un-repeated pass (official parity); on this benchmark's small generated inputs, sub-millisecond loop times make Beyond noisy at the per-task level. Aggregate Beyond@1 over 256 tasks is stable.
• —Execution runs under firejail rlimits, not the official in-process reliability guard.