reasoning-efficiency-authors/reasoning_efficiency

Reasoning Efficiency Evaluation Artifact

Anonymous review dataset accompanying the NeurIPS 2026 Evaluations & Datasets submission “Diagnosing Reasoning Efficiency with Trace-Optional Evaluation”.

The artifact contains benchmark instances, raw visible model outputs, token/count metadata, correctness and truncation flags, native workload metadata, derived model-level metrics, and decomposition tables used by the paper.

Files

• —instances/*.jsonl.gz: benchmark prompts, gold answers, and native workload metadata.
• —runs/*/part-*.jsonl.gz: per model-instance raw visible responses and scored run metadata.
• —derived/*.jsonl: model-level metrics, decomposition terms, and cross-benchmark tables.
• —sample/reviewer_sample.jsonl.gz: combined reviewer-inspection sample for the large raw-output release.
• —sample/*_reviewer_sample.jsonl.gz: benchmark-specific samples exposed in the Hugging Face viewer.
• —metadata/*.json: manifest, checksums, source summaries, model registry, and provenance.

Identifier semantics: raw run records use run_id for the model/configuration, so it repeats once per benchmark instance; run_instance_id is the unique raw run-row key. In model and decomposition tables, run_id is unique per row. reference_run_id and trace_reference_run_id name the table-level reference models and are intentionally repeated on decomposition rows.

Intended Use

Use this data to reproduce the paper’s reasoning-efficiency analyses or to test the trace-optional evaluation protocol on new model runs. The core metric layer requires only correctness, generated-token count, and truncation status.

Limitations

Generated visible tokens are a deployment-relevant proxy, not total FLOPs or latency. Nonpositive token counts are treated as unavailable. Native workload scales are benchmark-specific. Solver-derived workload depends on the declared reference pool. Trace diagnostics use visible-trace compression and do not validate semantic correctness of intermediate reasoning.