Master-AI-Lab/AtomWorldBench

AtomWorldBench

AtomWorldBench is a benchmark and dataset for evaluating the ability of Large Language Models (LLMs) and agents to perform 3D crystal structure manipulation from natural language instructions.

Given an input crystal structure in CIF format and a textual instruction, the model must generate the resulting crystal structure after applying the requested modification.

The dataset is released alongside the AtomWorld benchmark framework and is intended for:

• —Benchmarking spatial reasoning abilities of LLMs
• —Training structure-editing agents
• —Supervised fine-tuning (SFT)
• —Reinforcement learning and reward modeling research
• —Materials-science agent evaluation

Task Description

Each example contains:

{
  "action_prompt": "...",
  "input": "...",
  "output": "...",
  // ... other metadata
}

In addition, the repository provides a shared system_prompt.txt that can be used for every task.

where:

The task is:

Given (action_prompt, input), generate output.

The bench_data folder contains the data used in the AtomWorld Bench. Besides, we have generated ~5K data for each action, which can be used as training set.

Evaluation

AtomWorld uses structure-aware evaluation rather than text matching.

Typical verification steps include:

1. 1.CIF parsing
2. 2.Atom-count verification
3. 3.Structure matching
4. 4.RMSD calculation

For official evaluation and benchmarking tools, see the AtomWorld repository:

https://github.com/MasterAI-EAM/atomworld

Github Page:

https://masterai-eam.github.io/atomworld/

Repository Relationship

This Hugging Face repository contains the released datasets only.

The GitHub repository provides:

• —evaluation code
• —benchmark runner
• —dataset generation pipeline
• —API server for agent benchmarking
• —visualization and analysis utilities

Limitations

The dataset focuses on crystal-structure manipulation and does not directly evaluate:

• —materials-property prediction
• —electronic structure reasoning
• —synthesis planning
• —reaction prediction

Performance on AtomWorldBench should therefore be interpreted as a measure of structure-editing and spatial reasoning ability rather than general materials-science expertise.

Citation

If you use AtomWorldBench in your work, please cite:

@misc{lv2025atomworldbenchmarkevaluatingspatial,
      title={AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials},
      author={Taoyuze Lv and Alexander Chen and Fengyu Xie and Chu Wu and Jeffrey Meng and Dongzhan Zhou and Bram Hoex and Zhicheng Zhong and Tong Xie},
      year={2025},
      eprint={2510.04704},
      archivePrefix={arXiv},
      primaryClass={cond-mat.mtrl-sci}
}