LMucko/crystallography-open-database

Crystallography Open Database (COD) — Full Snapshot

A complete mirror of the Crystallography Open Database (COD) as a single Parquet file, combining all crystallographic metadata with the raw CIF file content in one queryable dataset.

Snapshot Details

Dataset Structure

One Parquet file (cod_full.parquet) with 75 columns — the 74 COD metadata fields plus a cif_text column containing the full raw CIF file content as a string.

Metadata Columns

CIF Text Column

Each CIF contains:

• —Unit cell parameters and standard uncertainties
• —Space group information (Hermann-Mauguin, Hall, IT number)
• —Atomic site coordinates (fractional) with displacement parameters
• —SHELX .res file (refinement details, atom lists, restraints)
• —SHELX .hkl file (observed structure factors — Fobs)
• —Publication metadata and COD database codes

Loading

from datasets import load_dataset

ds = load_dataset("your-username/cod", split="train")
print(ds[0]["formula"])      # e.g. "C16 H11 F N2 O"
print(ds[0]["sg"])           # e.g. "P 1 21/c 1"
print(ds[0]["cif_text"][:200])  # first 200 chars of the CIF

Or directly with PyArrow / DuckDB for out-of-core querying:

import pyarrow.parquet as pq

table = pq.read_table("cod_full.parquet", columns=["file", "formula", "sg", "year"])

SELECT file, formula, sg, year
FROM 'cod_full.parquet'
WHERE year >= 2020 AND sgNumber = 14
LIMIT 10;

Parsing CIF Text

For crystallographic analysis, parse cif_text with:

# gemmi (fastest, C++ bindings)
import gemmi
doc = gemmi.cif.read_string(row["cif_text"])
block = doc[0]

# pymatgen (full crystallography)
from pymatgen.core import Structure
import io
struct = Structure.from_str(row["cif_text"], fmt="cif")

Coverage

• —Earliest publication: 1915
• —Peak deposition years: 2012–2015 (~25–28K structures/year)
• —Recent deposits: ~10K/year
• —Compound types: organic, inorganic, metal-organic, minerals (excluding biopolymers)
• —Data sources: CCDC, AMCSD, IUCr journals, direct depositions

License

All data in the COD is dedicated to the public domain under CC0 1.0.

Users of the data should acknowledge the original authors of the structural data. The authors, title, journal, year, and doi columns are provided for this purpose.

Citation

If you use this dataset, please cite the COD:

Grazulis, S., Chateigner, D., Downs, R. T., Yokochi, A. F. T., Quirós, M., Lutterotti, L., Manakova, E., Butkus, J., Moeck, P. & Le Bail, A. (2009). Crystallography Open Database – an open-access collection of crystal structures. J. Appl. Cryst. 42, 726–729. doi:10.1107/S0021889809016690

Merkys, A., Vaitkus, A., Butkus, J., Okulič-Kazarinas, V., Kairys, V. & Gražulis, S. (2016). COD::CIF::Parser: an error-correcting CIF parser for the Perl language. J. Appl. Cryst. 49, 292–301. doi:10.1107/S1600576716000414

Source

• —Website: https://www.crystallography.net/
• —SQL database: sql.crystallography.net (user: cod_reader, database: cod)
• —Rsync: rsync://www.crystallography.net/cif/