xiaodu-ali/CrystalXRD-Bench

LLM4Mat-Bench: XRD Max-Peak HKL Identification Benchmark

Dataset Description

LLM4Mat-Bench is a multimodal benchmark for evaluating Vision-Language Models (VLMs) on crystallographic reasoning tasks. Given a theoretical X-ray Diffraction (XRD) pattern image and the corresponding crystal structure (CIF), the model must identify the Miller indices (HKL) of the crystallographic planes contributing to the highest-intensity peak.

This benchmark tests the intersection of visual pattern recognition and materials science domain knowledge — a challenging task that requires understanding both XRD physics and crystallographic indexing.

Task Definition

Input:

• —XRD pattern image (theoretical, Cu Kα radiation)
• —Chemical formula
• —Full CIF crystal structure data

Output:

• —Miller indices [h, k, l] (or [h, k, i, l] for hexagonal/trigonal systems) of all crystallographic planes contributing to the highest peak

Key challenge: A single visual peak may be an overlap of multiple Bragg reflections due to lattice symmetry or accidental degeneracy. The model must identify ALL contributing planes.

Dataset Statistics

Material Type Distribution

HKL Convention

Field Schema

XRD Pattern Generation Parameters

All XRD patterns are theoretical (computed from CIF, not experimental), generated with:

Ground Truth Method

The ground truth is generated using the "contributors union within tolerance" method:

1. 1.Compute all theoretical Bragg reflections from CIF using pymatgen
2. 2.Locate the highest-intensity peak position (θ_argmax) in the simulated pattern
3. 3.Find ALL reflections within ±0.30° of θ_argmax
4. 4.Exclude reflections with relative intensity < 10% of the strongest in the window
5. 5.The union of remaining HKL planes forms the ground truth set

Usage

from datasets import load_dataset

# Load the dataset
ds = load_dataset("<your-org>/LLM4Mat-Bench", split="test")

print(f"Total samples: {len(ds)}")
print(ds[0].keys())

# Access a sample
sample = ds[0]
print(f"Sample ID: {sample['sample_id']}")
print(f"Formula: {sample['formula']}")
print(f"GT HKLs: {sample['gt_hkls']}")
print(f"Image: {sample['image']}")  # PIL Image object

# Filter by dataset
import json
mp_samples = ds.filter(lambda x: x["dataset"] == "mp")
print(f"MP samples: {len(mp_samples)}")

# Parse ground truth
gt_hkls = json.loads(sample["gt_hkls"])
print(f"Ground truth HKL planes: {gt_hkls}")

Evaluation Metrics

We use set-based metrics since the task output is a set of HKL tuples:

• —Jaccard Similarity: |Pred ∩ GT| / |Pred ∪ GT|
• —Precision: |Pred ∩ GT| / |Pred|
• —Recall: |Pred ∩ GT| / |GT|
• —F1 Score: Harmonic mean of precision and recall

An evaluation script is provided: evaluate.py

python evaluate.py --predictions predictions.jsonl --ground_truth metadata.jsonl

Important Notes

• —[0,0,0] HKL tuples are treated as invalid and excluded from both predictions and ground truth
• —HKL equivalence is NOT applied: exact tuple matching only (e.g., [1,0,0] ≠ [-1,0,0])
• —XRD images show clean curves only — no peak annotations or markers
• —The properties field contains dataset-specific material properties as a JSON string

Citation

@misc{llm4matbench2025,
  title={LLM4Mat-Bench: A Multimodal Benchmark for Evaluating Vision-Language Models on Crystallographic XRD Analysis},
  year={2025},
  url={https://huggingface.co/datasets/<your-org>/LLM4Mat-Bench}
}

License

This dataset is released under the Creative Commons Attribution 4.0 International License (CC-BY-4.0).

The source crystallographic data originates from publicly available materials databases (Materials Project, JARVIS, OQMD, GNoME, etc.) under their respective licenses.