Anonymous-NeurIPS26-VeRA/VeRA

VeRA: Reasoning Benchmarks as Executable Specifications

NeurIPS 2026 - Evaluations & Datasets Track (Anonymous Submission)

What is VeRA?

Most reasoning benchmarks are static: the same problems are reused indefinitely, enabling memorization, format exploitation, and saturation. VeRA redefines a benchmark as an executable specification - a triple (template, generator, verifier) that can be sampled post-training to produce unlimited fresh instances with programmatically certified labels.

Once a specification is validated, labels for any newly sampled instance are computed by executing the verifier, so correctness amortizes across an unbounded stream of fresh variants at near-zero marginal cost.

Modes

VeRA operates in three complementary modes:

• —VeRA-E (equivalent): meaning-preserving rewrites that keep verifier semantics fixed. Used for robustness diagnostics and exposure detection.
• —VeRA-H (hardened): variants that systematically increase reasoning demand while remaining verifiable.
• —VeRA-H Pro: paired protocol that selects, per seed, the judge-ranked hardest verified variant from a fixed candidate pool.

On top of the automatic pipeline, we release *human-audited Verified subsets (220/220 VeRA-E items; 2,187/2,299 VeRA-H / VeRA-H Pro items at 95.13% retention), and pair-preserving Verified Full* subsets in which any rejected paired candidate is regenerated until audit passes.

Quick start

from datasets import load_dataset

# Recommended: human-audited Verified subsets
ds = load_dataset("Anonymous-NeurIPS26-VeRA/VeRA", "verah_pro_verified_aime2024")
for ex in ds["train"].select(range(2)):
    print(ex["variant_id"], ex["question"][:80], "->", ex["answer"])

# Original benchmark seeds
seeds = load_dataset("Anonymous-NeurIPS26-VeRA/VeRA", "seeds_aime2024")

# Raw VeRA-H pool (research / methodology reproduction)
raw = load_dataset("Anonymous-NeurIPS26-VeRA/VeRA", "verah_amobench")

Subsets

All subsets share the schema (benchmark, split, variant_id, question, answer) with benchmark and split providing self-describing provenance.

seeds/ - Original benchmark seeds (input to VeRA)

verae/ - VeRA-E - Verified-equivalent variants (raw automatic pool)

verah/ - VeRA-H - Verified-hardened variants (raw automatic pool)

verah_pro/ - VeRA-H Pro - Paired hardest verified variant per seed (raw automatic pool)

verah_verified/ - VeRA-H Verified - Human-audited subset (recommended for evaluation)

verah_pro_verified/ - VeRA-H Pro Verified - Human-audited paired subset (recommended for evaluation)

verah_verified_full/ - VeRA-H Verified Full - Pair-preserving regenerated-until-audit-passes subset

verah_pro_verified_full/ - VeRA-H Pro Verified Full - Pair-preserving Pro subset

Recommended use

• —For model comparison and leaderboard reporting, use the Verified (verah_verified* / verah_pro_verified*) and Verified Full releases.
• —For methodology reproduction (e.g., re-running the propose-execute-validate pipeline), use the raw automatic pools (verae, verah, verah_pro).
• —For familiarity / robustness diagnostics (the year-controlled AIME analysis), use the bilingual verae_aime2024 and verae_aime2025 subsets.

Provenance

Seeds are taken from publicly released benchmarks: GSM8K, AIME (1983-2001 / 2024 / 2025), Beyond-AIME, AMO-Bench, and GPQA-Diamond. Executable specifications are synthesized by a frontier Teacher LLM (GPT-5 default; Gemini 3 Pro and Claude 4.6 Opus used for cross-Teacher robustness controls) via a propose -> execute -> validate loop. Labels are produced by deterministic execution of the validated verifier - no LLM is in the labeling loop.

Full collection methodology, audit protocol, RAI metadata, and limitations are documented in croissant.json (Croissant 1.1) and in the paper appendix.

License

Apache-2.0. Seed benchmarks retain their original licenses; VeRA releases only newly-generated specifications and instances.

Anonymous code release

Code, executable specifications, sandbox runner, and synthesis pipeline: <https://anonymous.4open.science/r/VeRA-5EC9>