LMucko/crystallography-open-database

1---2license: cc0-1.03task_categories:4  - text-classification5  - feature-extraction6tags:7  - crystallography8  - chemistry9  - materials-science10  - CIF11  - crystal-structure12size_categories:13  - 100K<n<1M14---15 16# Crystallography Open Database (COD) — Full Snapshot17 18A complete mirror of the [Crystallography Open Database](https://www.crystallography.net/) (COD) as a single Parquet file, combining all crystallographic metadata with the raw CIF file content in one queryable dataset.19 20## Snapshot Details21 22| Field | Value |23|-------|-------|24| **Snapshot date** | 2026-07-06 |25| **Metadata fetched** | 2026-07-06 18:51 (UTC+2) — 533,486 entries |26| **CIF files downloaded** | 2026-07-06 18:34–21:58 — 533,862 files |27| **Total rows** | 533,486 (metadata) — 411 on-hold entries excluded |28| **File format** | Parquet (zstd compressed) |29| **License** | CC0 1.0 (Public Domain) |30 31## Dataset Structure32 33One Parquet file (`cod_full.parquet`) with 75 columns — the 74 COD metadata fields plus a `cif_text` column containing the full raw CIF file content as a string.34 35### Metadata Columns36 37| Column | Type | Description |38|--------|------|-------------|39| `file` | int64 | COD ID (7-digit identifier, e.g. `1553769`) |40| `a`, `b`, `c` | double | Unit cell lengths (Å) |41| `siga`, `sigb`, `sigc` | double | Standard uncertainties on cell lengths |42| `alpha`, `beta`, `gamma` | double | Unit cell angles (°) |43| `sigalpha`, `sigbeta`, `siggamma` | double | Standard uncertainties on cell angles |44| `vol` | double | Unit cell volume (ų) |45| `sigvol` | double | Standard uncertainty on volume |46| `celltemp` | double | Cell measurement temperature (K) |47| `sigcelltemp` | double | SU on cell temperature |48| `diffrtemp` | double | Diffraction measurement temperature (K) |49| `sigdiffrtemp` | double | SU on diffraction temperature |50| `cellpressure` | double | Cell measurement pressure (kPa) |51| `sigcellpressure` | double | SU on cell pressure |52| `diffrpressure` | double | Diffraction measurement pressure (kPa) |53| `sigdiffrpressure` | double | SU on diffraction pressure |54| `thermalhist` | string | Thermal history |55| `pressurehist` | string | Pressure history |56| `compoundsource` | string | Source of compound |57| `nel` | string | Number of distinct elements |58| `sg` | string | Space group (Hermann-Mauguin) |59| `sgHall` | string | Space group (Hall notation) |60| `sgNumber` | int64 | Space group number (1–230) |61| `commonname` | string | Common name |62| `chemname` | string | Chemical name |63| `mineral` | string | Mineral name |64| `formula` | string | Chemical formula (sum) |65| `calcformula` | string | Calculated formula |66| `cellformula` | string | Unit cell formula |67| `Z` | int64 | Number of formula units per unit cell |68| `Zprime` | double | Z′ (Z per asymmetric unit) |69| `acce_code` | string | CCDC accession code |70| `authors` | string | Publication authors |71| `title` | string | Publication title |72| `journal` | string | Journal name |73| `year` | int64 | Publication year |74| `volume` | int64 | Journal volume |75| `issue` | string | Journal issue |76| `firstpage` | string | First page |77| `lastpage` | string | Last page |78| `doi` | string | Digital Object Identifier |79| `method` | string | Structure determination method |80| `radiation` | string | Radiation type |81| `wavelength` | double | Wavelength (Å) |82| `radType` | string | Radiation type (detailed) |83| `radSymbol` | string | Radiation symbol |84| `Rall` | double | R-factor (all reflections) |85| `Robs` | double | R-factor (observed reflections) |86| `Rref` | double | R-factor (refined) |87| `wRall` | double | Weighted R-factor (all) |88| `wRobs` | double | Weighted R-factor (observed) |89| `wRref` | double | Weighted R-factor (refined) |90| `RFsqd` | double | R-factor on F² |91| `RI` | double | R-factor (intensity) |92| `gofall` | double | Goodness of fit (all) |93| `gofobs` | double | Goodness of fit (observed) |94| `gofgt` | double | Goodness of fit (I > 2σ(I)) |95| `gofref` | double | Goodness of fit (refined) |96| `duplicateof` | int64 | COD ID of which this is a duplicate |97| `optimal` | int64 | Flag for optimal structure |98| `status` | string | Status (e.g. `warnings`) |99| `flags` | string | Status flags |100| `text` | string | Concatenated bibliography text |101| `svnrevision` | int64 | SVN revision of the COD entry |102| `date` | string | Deposition date (`YYYY-MM-DD`) |103| `time` | string | Deposition time (`HH:MM:SS`) |104| `onhold` | string | On-hold release date (empty if released) |105 106### CIF Text Column107 108| Column | Type | Description |109|--------|------|-------------|110| `cif_text` | string | Full raw CIF file content (text) |111 112Each CIF contains:113- Unit cell parameters and standard uncertainties114- Space group information (Hermann-Mauguin, Hall, IT number)115- Atomic site coordinates (fractional) with displacement parameters116- SHELX `.res` file (refinement details, atom lists, restraints)117- SHELX `.hkl` file (observed structure factors — Fobs)118- Publication metadata and COD database codes119 120## Loading121 122```python123from datasets import load_dataset124 125ds = load_dataset("your-username/cod", split="train")126print(ds[0]["formula"])      # e.g. "C16 H11 F N2 O"127print(ds[0]["sg"])           # e.g. "P 1 21/c 1"128print(ds[0]["cif_text"][:200])  # first 200 chars of the CIF129```130 131Or directly with PyArrow / DuckDB for out-of-core querying:132 133```python134import pyarrow.parquet as pq135 136table = pq.read_table("cod_full.parquet", columns=["file", "formula", "sg", "year"])137```138 139```sql140SELECT file, formula, sg, year141FROM 'cod_full.parquet'142WHERE year >= 2020 AND sgNumber = 14143LIMIT 10;144```145 146## Parsing CIF Text147 148For crystallographic analysis, parse `cif_text` with:149 150```python151# gemmi (fastest, C++ bindings)152import gemmi153doc = gemmi.cif.read_string(row["cif_text"])154block = doc[0]155 156# pymatgen (full crystallography)157from pymatgen.core import Structure158import io159struct = Structure.from_str(row["cif_text"], fmt="cif")160```161 162## Coverage163 164- **Earliest publication**: 1915165- **Peak deposition years**: 2012–2015 (~25–28K structures/year)166- **Recent deposits**: ~10K/year167- **Compound types**: organic, inorganic, metal-organic, minerals (excluding biopolymers)168- **Data sources**: CCDC, AMCSD, IUCr journals, direct depositions169 170## License171 172All data in the COD is dedicated to the public domain under [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/).173 174Users of the data should acknowledge the original authors of the structural data. The `authors`, `title`, `journal`, `year`, and `doi` columns are provided for this purpose.175 176## Citation177 178If you use this dataset, please cite the COD:179 180> Grazulis, S., Chateigner, D., Downs, R. T., Yokochi, A. F. T., Quirós, M., Lutterotti, L., Manakova, E., Butkus, J., Moeck, P. & Le Bail, A. (2009). *Crystallography Open Database – an open-access collection of crystal structures.* J. Appl. Cryst. 42, 726–729. doi:10.1107/S0021889809016690181 182> Merkys, A., Vaitkus, A., Butkus, J., Okulič-Kazarinas, V., Kairys, V. & Gražulis, S. (2016). *COD::CIF::Parser: an error-correcting CIF parser for the Perl language.* J. Appl. Cryst. 49, 292–301. doi:10.1107/S1600576716000414183 184## Source185 186- **Website**: https://www.crystallography.net/187- **SQL database**: `sql.crystallography.net` (user: `cod_reader`, database: `cod`)188- **Rsync**: `rsync://www.crystallography.net/cif/`189