Master-AI-Lab/AtomWorldBench

1---2license: mit3task_categories:4- question-answering5language:6- en7tags:8- agent9- materials_science10- spatial_reasoning11- action12pretty_name: AtomMotor13size_categories:14- 1K<n<10K15configs:16- config_name: default17  data_files:18  - split: train19    path: "train/*.json"20  - split: bench21    path: "bench/*.json"22---23 24# AtomWorldBench25 26AtomWorldBench is a benchmark and dataset for evaluating the ability of Large Language Models (LLMs) and agents to perform **3D crystal structure manipulation** from natural language instructions.27 28Given an input crystal structure in CIF format and a textual instruction, the model must generate the resulting crystal structure after applying the requested modification.29 30The dataset is released alongside the AtomWorld benchmark framework and is intended for:31 32- Benchmarking spatial reasoning abilities of LLMs33- Training structure-editing agents34- Supervised fine-tuning (SFT)35- Reinforcement learning and reward modeling research36- Materials-science agent evaluation37 38## Task Description39 40Each example contains:41 42```json43{44  "action_prompt": "...",45  "input": "...",46  "output": "...",47  // ... other metadata48}49```50 51In addition, the repository provides a shared `system_prompt.txt` that can be used for every task.52 53where:54 55| Field | Description |56|---------|-------------|57| `action_prompt` | Natural-language instruction describing the required structure modification |58| `input` | Input crystal structure in CIF format |59| `output` | Ground-truth crystal structure after applying the instruction |60 61The task is:62 63> Given (`action_prompt`, `input`), generate `output`.64 65 66The `bench_data` folder contains the data used in the AtomWorld Bench. Besides, we have generated ~5K data for each action, which can be used as training set.67 68 69## Evaluation70 71AtomWorld uses structure-aware evaluation rather than text matching.72 73Typical verification steps include:74 751. CIF parsing762. Atom-count verification773. Structure matching784. RMSD calculation79 80For official evaluation and benchmarking tools, see the AtomWorld repository:81 82https://github.com/MasterAI-EAM/atomworld83 84Github Page:85 86https://masterai-eam.github.io/atomworld/87 88## Repository Relationship89 90This Hugging Face repository contains the released datasets only.91 92The GitHub repository provides:93 94- evaluation code95- benchmark runner96- dataset generation pipeline97- API server for agent benchmarking98- visualization and analysis utilities99 100## Limitations101 102The dataset focuses on crystal-structure manipulation and does not directly evaluate:103 104- materials-property prediction105- electronic structure reasoning106- synthesis planning107- reaction prediction108 109Performance on AtomWorldBench should therefore be interpreted as a measure of structure-editing and spatial reasoning ability rather than general materials-science expertise.110 111## Citation112 113If you use AtomWorldBench in your work, please cite:114 115```bibtex116@misc{lv2025atomworldbenchmarkevaluatingspatial,117      title={AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials},118      author={Taoyuze Lv and Alexander Chen and Fengyu Xie and Chu Wu and Jeffrey Meng and Dongzhan Zhou and Bram Hoex and Zhicheng Zhong and Tong Xie},119      year={2025},120      eprint={2510.04704},121      archivePrefix={arXiv},122      primaryClass={cond-mat.mtrl-sci}123}