CoolFace
Datasetpublic

FrontisAI/NatureBench-Harbor

NatureBench Tasks for Harbor This dataset provides the 90 NatureBench tasks prebuilt for evaluation with Harbor. The Harbor packaging keeps the original data, evaluation protocol and validity-judge policy unchanged. 📄 arXiv paper: https://arxiv.org/abs/2606.24530 💻 GitHub code repository: https://github.com/FrontisAI/NatureBench ⚓ Harbor instructions: https://github.com/FrontisAI/NatureBench/tree/main/harbor 📦 Original NatureBench packages:… See the full description on the dataset page: https://huggingface.co/datasets/FrontisAI/NatureBench-Harbor.

sourceHugging Faceotherupdated 26d agoView on Hugging Face
0likes296downloads
Dataset Card

NatureBench Tasks for Harbor

<!-- Harbor-compatible task packages for NatureBench. -->

This dataset provides the 90 NatureBench tasks prebuilt for evaluation with Harbor. The Harbor packaging keeps the original data, evaluation protocol and validity-judge policy unchanged.

  • —📄 arXiv paper: <https://arxiv.org/abs/2606.24530>
  • —💻 GitHub code repository: <https://github.com/FrontisAI/NatureBench>
  • —⚓ Harbor instructions: <https://github.com/FrontisAI/NatureBench/tree/main/harbor>
  • —📦 Original NatureBench packages: <https://huggingface.co/datasets/FrontisAI/NatureBench>
  • —🏆 Leaderboard: <https://frontisai.github.io/NatureBench/>

Harbor Support

NatureBench evaluates coding agents on scientific machine-learning problems drawn from Nature-family papers. This repository distributes the rendered Harbor task packages.

The conversion adapter, runtime extensions, official downloader, and reference run configuration are maintained in the NatureBench GitHub repository.

Harbor Task Packages

Most tasks are distributed as task_archives/<task-id>.tar.gz. Very large tasks are split into multiple archives under task_archives/<task-id>/; the official downloader handles both layouts. The extracted task has the following structure:

text
<task-id>/
├── task.toml
├── instruction.md
├── licenses/
├── environment/
│   ├── Dockerfile
│   ├── docker-compose.yaml
│   ├── input/
│   └── sidecar/
└── tests/
    ├── Dockerfile
    ├── test.sh
    ├── verifier.py
    └── context/
ComponentPurpose
task.tomlHarbor task configuration.
instruction.mdNatureBench prompt presented to the agent.
licenses/Third-party data licenses for the task.
environment/Agent container, task inputs, and evaluation sidecar.
tests/Separate verifier that finalizes the score and applies the validity judge.

See the Harbor guide for more details.

How to Download and Use

NatureBench Harbor tasks are used with the companion code in the NatureBench GitHub repository. The official downloader below downloads and extracts the task packages automatically. If you download the data directly from Hugging Face, extract the task archive (task_archives/<task-id>.tar.gz) or all of its archive parts (every archive under task_archives/<task-id>/) before running Harbor.

bash
git clone https://github.com/FrontisAI/NatureBench.git
cd NatureBench/harbor

# Download all 90 tasks
python scripts/download_tasks.py --output-dir ./tasks

# Download selected tasks
python scripts/download_tasks.py \
  --output-dir ./tasks \
  --task-ids s43588-024-00689-2 s42256-024-00833-7

# Download one or more compute groups
python scripts/download_tasks.py \
  --output-dir ./tasks \
  --compute gpu_low gpu_high

After downloading, follow the Harbor guide to set up and run an evaluation.

License

The top-level LICENSE applies only to original NatureBench contributions. Third-party data is governed by the notices in each task's <task-id>/licenses/.

Citation

If you use NatureBench in your research, please cite:

bibtex
@misc{wang2026naturebench,
  title         = {NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?},
  author        = {Yuru Wang and Lejun Cheng and Yuxin Zuo and Sihang Zeng and Bingxiang He and Che Jiang and Junlin Yang and Yuchong Wang and Kaikai Zhao and Weifeng Huang and Kai Tian and Zhenzhao Yuan and Jincheng Zhong and Weizhi Wang and Ning Ding and Bowen Zhou and Kaiyan Zhang},
  year          = {2026},
  eprint        = {2606.24530},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.24530}
}