ProCreations/repro-contextual-rollout-bandits-for-reinforcement-learning-with-verifiable-rewards

FormulaCode released-artifact reproduction

This semantic-v4 package evaluates all six registered claims for FormulaCode: Evaluating Agentic Optimization on Large Codebases (OpenReview, arXiv v1). It pins the authors' official evaluation harness, website data repository, and living Hugging Face dataset.

Run the deterministic audit twice:

PYTHONDONTWRITEBYTECODE=1 PYTHONHASHSEED=0 PYTHONWARNINGS=error \
  python3 -W error native_formulacode_audit.py --out outputs
PYTHONDONTWRITEBYTECODE=1 PYTHONHASHSEED=0 PYTHONWARNINGS=error \
  python3 -W error native_formulacode_audit.py --out packaged_replay
python3 validate_evidence.py

The audit consumes all 5,142 rows of the official website CSV, all 1,215 unique tasks in the pinned current dataset, all eight non-human global leaderboard rows, all 24 optimization-scale cells, all 39 non-null popularity cells, and all 71 non-null strategy cells. The paired outputs are byte-identical.

Two claims are verified and four are literal falsifications. Claims 1 and 2 use the first-party current dataset contradiction (1,215 rather than the registered present-tense 957). Claim 4's named OpenHands patterns hold, but its blanket function-level superiority does not. Claim 5 remains non-monotonic with Q2 best and Q4 worst, but the current Q1 mean is -0.034623 and rounds to -0.03, not the exact registered -0.04. Historical camera-ready statistics are not disputed or rewritten.

The scheduled target is ProCreations/repro-contextual-rollout-bandits-for-reinforcement-learning-with-verifiable-rewards. That slug was precreated as a private untagged quota-safety shell before the scientific replacement was selected. The mismatch is intentionally disclosed; the package metadata and sole paper tag are FormulaCode paper-WArbqRUsAe. No remote mutation has been performed while the campaign guard is closed.