songjhPKU/RxnOptBench

RxnOptBench

RxnOptBench is a fixed benchmark release for offline evaluation of reaction-condition recommendation systems. It packages the current testsetV6 benchmark as a Hugging Face dataset repository that is compatible with the Dataset Viewer and Croissant metadata generation workflow required by the NeurIPS 2026 Evaluations and Datasets hosting guidelines.

This review release was generated on 2026-05-03T15:52:23+00:00 with release version v6-neurips2026-review. It contains 4773 benchmark questions from 552 structured source tables and 236 unique source-article identifiers carried in meta.doi.

Configs

Each config has a single test split.

Repository Layout

• —data/jsonl/: canonical upload-side data files used by the Hugging Face Dataset Viewer.
• —data/original_json/: original benchmark JSON files preserved in their native evaluation format.
• —prompts/: prompt templates used to render benchmark questions for model evaluation.
• —examples/: example prediction files that demonstrate the required submission formats.
• —evaluation/: reference evaluation scripts and shared helpers.
• —metadata/: release manifests, source provenance manifests, and the Croissant RAI patch file for NeurIPS submission.
• —docs/: source build reports and submission notes.

Loading

from datasets import load_dataset

ds = load_dataset("YOUR_NAMESPACE/RxnOptBench", "multiple_choice_single_varying", split="test")
print(ds[0]["id"])

The JSONL files preserve the nested benchmark structure instead of flattening it. This keeps the Hugging Face-hosted representation close to the original evaluation data while remaining compatible with the Dataset Viewer and Croissant generation flow.

Evaluation

• —Multiple-choice predictions must be JSON maps from id to a single 0-based option index.
• —Fill-in-the-blank predictions must map each id to a {condition_key: candidate_index} dictionary.
• —Use the scripts in evaluation/ together with the original JSONs in data/original_json/ if you want byte-for-byte compatibility with the current benchmark workflow.

Provenance

The release is derived from the current structured benchmark outputs:

• —data/testsetV6/20260502_000057_multiple_choice
• —data/testsetV6/20260502_000115_fill_in_blank

Additional provenance manifests are provided in:

• —metadata/source_articles.jsonl
• —metadata/source_tables.jsonl
• —metadata/release_manifest.json
• —docs/source_report_multiple_choice.md
• —docs/source_report_fill_in_blank.md

Intended Use

This benchmark is intended for:

• —Offline evaluation of reaction-condition recommendation systems.
• —Controlled comparison of model behavior across multiple-choice, reference-augmented, and fill-in-the-blank condition-selection tasks.
• —Ablation studies on the effect of reference conditions, candidate-space construction, and scoring rules on model performance.

Out-of-Scope Use

This benchmark is not validated for:

• —Direct laboratory deployment without expert review.
• —Autonomous or safety-critical decision making.
• —Open-ended reaction planning beyond the provided candidate spaces.
• —Estimating absolute reaction success outside the experimentally observed combinations represented in the source tables.

Responsible AI Notes

Limitations and Biases

• —This benchmark measures offline selection or ranking of literature-observed reaction conditions under constrained candidate spaces; it does not measure end-to-end laboratory execution, safety, cost, scalability, or open-ended planning.
• —Only structured source tables that passed parsing and filtering requirements were retained. Papers with ambiguous condition fields, missing SMILES, missing usable yields, or non-tabular reporting styles are under-represented.
• —Scores are relative to the best available option or legal combination inside each source-table-derived question, not to a global optimum across the full chemical search space.

• —The benchmark is literature-derived and therefore inherits publication bias toward successful, interesting, or publishable reactions.
• —Coverage is uneven across reaction classes, substrates, condition categories, journals, publishers, and publication years because the source pool depends on the upstream structured-table extraction pipeline.
• —Reference variants intentionally expose low- to mid-performing conditions while withholding the best condition. This is a deliberate evaluation design and not a natural distribution of auxiliary evidence.

Personal or Sensitive Information

No personal, demographic, medical, political, or otherwise human-sensitive attributes are intentionally collected. Records describe chemical reactions, reaction conditions, observed outcomes, and normalized source-article identifiers from published literature tables.

Social Impact

Standardized chemistry benchmarks can improve reproducibility and make model comparisons easier. At the same time, benchmark scores can be over-interpreted as evidence of laboratory readiness, and literature-derived benchmarks can propagate existing publication and reporting biases. This release therefore includes explicit provenance manifests, source reports, evaluation scripts, and scope limitations.

Synthetic Data

This benchmark release does not intentionally include synthetic reaction records.

Croissant and NeurIPS Submission

After the repository is uploaded to Hugging Face and the Dataset Viewer is available:

1. 1.Retrieve the auto-generated core Croissant metadata from the Hugging Face dataset page or /api/datasets/<repo_id>/croissant.
2. 2.Merge the patch file at metadata/croissant_rai_patch.json into the core Croissant JSON.
3. 3.Validate the final Croissant JSON with the NeurIPS-recommended validator before submitting it to OpenReview.

The helper script used locally for step 2 is tools/export_neurips_croissant.py in the source workspace that generated this repository.

License

The current repository template uses the Hugging Face metadata tag other. Before public release, the maintainers should replace it with the final redistribution license and update LICENSE.md accordingly.

Citation

Update this section with the final paper citation or dataset citation before public release.