LeMaterial/LeMat-Bulk

1---2dataset_info:3- config_name: compatible_pbe4  features:5  - name: elements6    sequence: string7  - name: nsites8    dtype: int329  - name: chemical_formula_anonymous10    dtype: string11  - name: chemical_formula_reduced12    dtype: string13  - name: chemical_formula_descriptive14    dtype: string15  - name: nelements16    dtype: int817  - name: dimension_types18    sequence: int819  - name: nperiodic_dimensions20    dtype: int821  - name: lattice_vectors22    sequence:23      sequence: float6424  - name: immutable_id25    dtype: string26  - name: cartesian_site_positions27    sequence:28      sequence: float6429  - name: species30    dtype: string31  - name: species_at_sites32    sequence: string33  - name: last_modified34    dtype: string35  - name: elements_ratios36    sequence: float6437  - name: stress_tensor38    sequence:39      sequence: float6440  - name: energy41    dtype: float6442  - name: magnetic_moments43    sequence: float6444  - name: forces45    sequence:46      sequence: float6447  - name: total_magnetization48    dtype: float6449  - name: dos_ef50    dtype: float6451  - name: functional52    dtype: string53  - name: cross_compatibility54    dtype: bool55  - name: entalpic_fingerprint56    dtype: string57  splits:58  - name: train59    num_bytes: 804354244460    num_examples: 533529961  download_size: 304745915162  dataset_size: 804354244463- config_name: compatible_pbesol64  features:65  - name: elements66    sequence: string67  - name: nsites68    dtype: int3269  - name: chemical_formula_anonymous70    dtype: string71  - name: chemical_formula_reduced72    dtype: string73  - name: chemical_formula_descriptive74    dtype: string75  - name: nelements76    dtype: int877  - name: dimension_types78    sequence: int879  - name: nperiodic_dimensions80    dtype: int881  - name: lattice_vectors82    sequence:83      sequence: float6484  - name: immutable_id85    dtype: string86  - name: cartesian_site_positions87    sequence:88      sequence: float6489  - name: species90    dtype: string91  - name: species_at_sites92    sequence: string93  - name: last_modified94    dtype: string95  - name: elements_ratios96    sequence: float6497  - name: stress_tensor98    sequence:99      sequence: float64100  - name: energy101    dtype: float64102  - name: magnetic_moments103    sequence: float64104  - name: forces105    sequence:106      sequence: float64107  - name: total_magnetization108    dtype: float64109  - name: dos_ef110    dtype: float64111  - name: functional112    dtype: string113  - name: cross_compatibility114    dtype: bool115  - name: entalpic_fingerprint116    dtype: string117  splits:118  - name: train119    num_bytes: 646300349120    num_examples: 447824121  download_size: 231942286122  dataset_size: 646300349123- config_name: compatible_scan124  features:125  - name: elements126    sequence: string127  - name: nsites128    dtype: int32129  - name: chemical_formula_anonymous130    dtype: string131  - name: chemical_formula_reduced132    dtype: string133  - name: chemical_formula_descriptive134    dtype: string135  - name: nelements136    dtype: int8137  - name: dimension_types138    sequence: int8139  - name: nperiodic_dimensions140    dtype: int8141  - name: lattice_vectors142    sequence:143      sequence: float64144  - name: immutable_id145    dtype: string146  - name: cartesian_site_positions147    sequence:148      sequence: float64149  - name: species150    dtype: string151  - name: species_at_sites152    sequence: string153  - name: last_modified154    dtype: string155  - name: elements_ratios156    sequence: float64157  - name: stress_tensor158    sequence:159      sequence: float64160  - name: energy161    dtype: float64162  - name: magnetic_moments163    sequence: float64164  - name: forces165    sequence:166      sequence: float64167  - name: total_magnetization168    dtype: float64169  - name: dos_ef170    dtype: float64171  - name: functional172    dtype: string173  - name: cross_compatibility174    dtype: bool175  - name: entalpic_fingerprint176    dtype: string177  splits:178  - name: train179    num_bytes: 597846818180    num_examples: 422840181  download_size: 208628422182  dataset_size: 597846818183- config_name: non_compatible184  features:185  - name: elements186    sequence: string187  - name: nsites188    dtype: int32189  - name: chemical_formula_anonymous190    dtype: string191  - name: chemical_formula_reduced192    dtype: string193  - name: chemical_formula_descriptive194    dtype: string195  - name: nelements196    dtype: int8197  - name: dimension_types198    sequence: int8199  - name: nperiodic_dimensions200    dtype: int8201  - name: lattice_vectors202    sequence:203      sequence: float64204  - name: immutable_id205    dtype: string206  - name: cartesian_site_positions207    sequence:208      sequence: float64209  - name: species210    dtype: string211  - name: species_at_sites212    sequence: string213  - name: last_modified214    dtype: string215  - name: elements_ratios216    sequence: float64217  - name: stress_tensor218    sequence:219      sequence: float64220  - name: energy221    dtype: float64222  - name: magnetic_moments223    sequence: float64224  - name: forces225    sequence:226      sequence: float64227  - name: total_magnetization228    dtype: float64229  - name: dos_ef230    dtype: float64231  - name: functional232    dtype: string233  - name: cross_compatibility234    dtype: bool235  - name: entalpic_fingerprint236    dtype: string237  splits:238  - name: train239    num_bytes: 818845899240    num_examples: 519627241  download_size: 270108882242  dataset_size: 818845899243configs:244- config_name: compatible_pbe245  data_files:246  - split: train247    path: compatible_pbe/train-*248- config_name: compatible_pbesol249  data_files:250  - split: train251    path: compatible_pbesol/train-*252- config_name: compatible_scan253  data_files:254  - split: train255    path: compatible_scan/train-*256- config_name: non_compatible257  data_files:258  - split: train259    path: non_compatible/train-*260license: cc-by-4.0261tags:262- chemistry263size_categories:264- 1M<n<10M265pretty_name: LeMatBulk266---267 268 269## Dataset Description270 271- **Homepage:** https://www.lematerial.org/272- **Repository:** https://github.com/lematerial/lematerial273- **Point of Contact:** contact@lematerial.org274 275 276**Motivation**: check out the blog post [https://huggingface.co/blog/lematerial](https://huggingface.co/blog/lematerial) to hear more about the motivation behind the creation of this dataset.277 278## Changelog:2792025.04.17 (hash: NOT YET  RELEASED):280- We have changed the Yb default pseudopotential to Yb_3 from VASP, this is the same that Materials Project uses. In the previous version we had kept it as Yb, and Materials Project had to Yb-containing materials. Alexandria and OQMD uses Yb. As a result no Yb-containing materials are cross_compatible=True in the latest iteration.281- We have filtered out high-force materials (norm of forces > 0.1eV/A). The criteria is still quite large. This mostly affect Materials Project materials.282- We have pulled the latest materials from Alexandria and Materials Project, as a result there are more materials in LeMat-Bulk and LeMat-Bulk-Unique283- We have added energy_corrected field for the PBE subset using the Materials Project 2020 compatibility scheme across the entire dataset284- We have added the international spacegroup number using Optimade's space_group_it_number and computed using Moyopy285- We have re-computed the hash using the BAWL fingerprint with the spacegroup processed by Moyopy and the entalpic_fingerprint has been changed to bawl_fingerprint. The fingerprint can be found in our newly released [lematerial-hasher libary](https://github.com/LeMaterial/lematerial-hasher)286- We have added r2SCAN data from Materials Project287- We used logic as defined in Materials Project's emmet package to select which MP task to include as the task representing the bulk material for MP materials.288- We added certain validation tests, as defined in our newly released [lematerial-fetcher library](https://github.com/LeMaterial/lematerial-fetcher)289 290Initial release, Dec 2024 (hash: 704b431aa574902f423b1c28f9b90c358d8cd1e1)291 292 293## Download and use within Python294```python295from datasets import load_dataset296 297dataset = load_dataset('LeMaterial/LeMat-Bulk', 'compatible_pbe')298 299# to use a specific database revision, based on the commit hash:300dataset = load_dataset('LeMaterial/LeMat-Bulk', 'compatible_pbe', revision='704b431aa574902f423b1c28f9b90c358d8cd1e1')301 302# convert to Pandas, if you prefer working with this type of object:303df = dataset['train'].to_pandas()304```305 306## Data fields307 308| **Feature name** | **Data type** | **Description** | **[Optimade](https://www.optimade.org/optimade) required field** |309| --- | --- | --- | --- |310| **elements** | Sequence[String] | A list of elements in the structure. For example a structure with composition Li2O7 will have `[”Li”,”O”]` in its elements.  Notes: Currently not necessarily sorted but future iteration will be sorted by alphabetic order. | ✅ |311| **nsites** | Integer | The total number of sites in the structure. For example a structure with an un-reduced composition of Li4O2 will have a total of `6` sites. | ✅ |312| **chemical_formula_anonymous** | String | Anonymous formula for a chemical structure, sorted by largest contributing species, and reduced by greatest common divisor. For example a structure with a O2Li4 un-reduced composition will have a anonymous formula of `A2B`. “1”’s at the end of an element composition are dropped (ie not A2B1) | ✅ |313| **chemical_formula_reduced** | String | Reduced by the greatest common divisor chemical composition. For example a structure with a un-reduced composition of O2Li4 will have a reduced composition of `Li2O`. Elements with a reduced composition of 1 have the “1” dropped. Elements are sorted by alphabetic ordering. Notes: Not using the same method of Pymatgen’s composition reduction method which takes into account certain elements existing in diatomic states. | ✅ |314| **chemical_formula_descriptive** | String | A more descriptive chemical formula for the structure, for example a fictive structure of a 6-fold hydrated Na ion might have a descriptive chemical formula of Na(H2O)6, or a Titanium chloride organic dimer might have a descriptive formula of [(C5H5)2TiCl]2. Note: this field is absolutely not standardized across the database. Where possible if available we scrapped as is from the respective databases. Where not possible this may be the same as the chemical formula reduced. | ✅ Note: not standardized in naming approach. |315| **space_group_it_number** | Integer | The international space group of the bulk structure as computed by Moyopy | ✅ |316| **nelements** | Integer | Total number of different elements in a structure. For example Li4O2 has only `2` separate elements. | ✅ |317| **dimension_types** | Sequence[Integer], shape = 3x1 | Periodic boundary conditions for a given structure. Because all of our materials are bulk materials for this database it is `[1, 1, 1]`, meaning it is periodic in x, y, and z dimensions. | ✅ |318| **nperiodic_dimensions** | Integer | The number of repeating periodic boundary conditions, because all our structures in this database are bulk structures, they are repeating in x, y, and z dimensions and thus they have `3` periodic dimensions. | ✅ |319| **lattice_vectors** | Sequence[Sequence[Floats]], shape = 3x3 | The matrix of the structures. For example a cubic system with a lattice a=4.5 will have a `[[4.5,0,0],[0,4.5,0],[0,0,4.5]]` lattice vector entry. | ✅ |320| **immutable_id** | String | The material ID associated with the structure from the respective database. Note: OQMD IDs are simply integers, thus we converted them to be “oqmd-YYY” | ✅ |321| **cartesian_site_positions** | Sequence[Sequence[Floats]], shape = Nx3 | In cartesian units (not fractional units) the coordinates of the species. These match the ordering of all site based properties such as `species_at_sites`, `magneitc_moments` and `forces`. For example a material with a single element placed at a fractional coordinate of [0.5, 0.5, 0.5] with a cubic lattice with a=2, will have a cartesian_site_positions of `[1, 1, 1]`. | ✅ |322| **species** | JSON | An [Optimade](https://www.optimade.org/optimade) field that includes information about the species themselves, such as their mass, their name, their labels, etc. Note: we have not currently filled out the mass portion of the species. Additionally, none of our inputted structures should be solid solution thus the on-site concentration for all our species should be [1]. This is an Optimade field. | ✅ |323| **species_at_sites** | Sequence[String] | An array of the chemical elements belonging to each site, for example a structure with an un-reduced composition of Li2O2 may have an entry of `[”Li”, “Li”, “O”, “O”]` for this field, where each species should match the other site based properties such as `cartesian_site_positions`. | ✅ |324| **last_modified** | Date/time | The date that the entry was last modified from the respective database it was pulled from. Note: we could not find this information in OQMD so we used the date of the latest database release as the input for this field. | ✅ |325| **elements_ratios** | Dictionary | The fractional composition for a given structure in dictionary format. For example a structure with an unreduced composition of Li2O4 would have an entry of `{’Li’:0.3333, ‘O’:0.6667}` | ✅ |326| **stress_tensor** | Sequence[Sequence[Float]], shape = 3x3 | The full 3x3 vector for stress tensor in units of kB. Note: for OQMD stress tensor were given in Voigt notation, and were converted to the full tensor. |  |327| **energy** | Float | The uncorrected energy from VASP in eV. |  |328| **magnetic_moments** | Sequence[Floats] | The magnetic moment per site given in µB. |  |329| **forces** | Sequence[Sequence[Floats]], shape = 3xN | The force per site, in the proper order of the sites based on other site specific fields for each site in the x, y and z directions, given in eV/A. |  |330| **total_magnetization** | Float | The total magnetization of the structure in µB. Note: the sum of the magnetic moments is not always the total magnetization of the structure reported. |  |331| **functional** | String, either ‘pbe’, ‘pbesol’ or ‘scan’ | What functional was used to calculate the data point in the row. |  |332| **cross_compatibility** | Boolean | Whether or not this data can be mixed with other rows from a DFT calculation parameter perspective. More information on our approach below. |  |333| **bawl_fingerprint** (previously entalpic_fingerprint) | String | Results of initial version of materials fingerprint function as described in [blogpost]. Code release to come soon| |334 335## Available subsets336 337To better support the diverse communities that may utilize this dataset, we are providing the following subsets of our database:338 339- **Compatible, PBE (default)**: This subset includes rows filtered to ensure cross-compatibility from a DFT perspective. For details on the filtering methodology, see the section below. Only PBE records are included. We designate this as the default subset to prevent accidental training of models on non-compatible data.340- **Compatible, PBESol**: Similar to the Compatible, PBE subset, but includes only PBESol data.341- **Compatible, SCAN**: Similar to the Compatible, PBE subset, but includes only SCAN data.342- **All**: This includes all records formatted as described above. **Disclaimer**: Researchers must carefully evaluate the suitability of individual rows for their specific applications.343 344 345| **Database** | **Number of materials** | **Number of structures*** |346| --- | --- | --- |347| Materials Project | 148,453 | 189,403 |348| Alexandria | 4,635,066 | 5,459,260 |349| OQMD | 1,076,926 | 1,076,926 |350| LeMaterial (All) | 5,860,446 | 6,725,590 | 351| LeMaterial (Compatible, PBE) | 5,335,299 | 5,335,299 |352| LeMaterial (Compatible, PBESOL) | 447,824 |  447,824 |353| LeMaterial (Compatible, SCAN) | 422,840 | 422,840 |354 355 356***Number of structures**: only includes the output of resulting calculations from either a structure optimization for any available functional. For MP we do not consider all of their structures from the relaxation trajectory for instance, nor from tasks that are not structure optimization. For OQMD we only consider the output of structure relaxation as well, not accounting for any other calculations they performed.357 358## Method for compatibility compliance359 360To ensure compatibility of rows from a DFT perspective, we implemented the following compatibility scheme:361 362- **Pseudopotentials**: Calculations were verified to use consistent pseudopotentials. Notably, most pseudopotentials were aligned between MP and Alexandria, except for vanadium (where Alexandria used V_sv and MP used V_pv) and cesium (where Alexandria used a later version of the generic pseudopotential). For OQMD, this resulted in incompatibilities across records involving the following elements: `Ca, Ti, V, Cr, Mn, Ru, Rh, Ce, Eu, Yb`. We note that at the time of this release Materials Project deprecated all Yb containing materials due to the use of a pseudopotential that led to different than expected results. Thus no Yb containing materials from MP are in our database.363- **Hubbard U Parameters**: To ensure uniformity in Hubbard U parameters, we excluded records containing oxygen (O) and any of the following elements: `V, Cr, Mn, Fe, Ni, Cu, Th, U, Np, Pu, Mo, W`. Similarly, records containing fluorine (F) and any of the following elements: Co, Cr, Fe, Mn, Mo, Ni, V, W were also excluded. This exclusion applied specifically to OQMD, which used different Hubbard U parameters compared to MP and Alexandria. However, records from OQMD containing `O` and `Co` were retained, as their Hubbard U parameter differed by only 0.02 eV.364- **Spin Polarization**: OQMD only considered spin-polarized calculations for structures with d or f electrons. While non-spin-polarized calculations are not inherently incompatible (as they represent higher-energy magnetic phases compared to the ground state), we decided to exclude non-spin-polarized calculations for this release. This led to the removal of structures containing only the following elements: `H, Li, Be, Na, Mg, K, Ca, Rb, Sr, Cs, Ba, Fr, Ra, B, C, N, O, F, Ne, He, Al, Si, P, S, Cl, Ar, Ga, Ge, As, Se, Br, Kr, In, Sn, Sb, Te, I, Xe, Tl, Pb, Bi, Po, At, Rn` from OQMD.365- **Convergence Criteria**: OQMD typically used a larger plane-wave cutoff but a less dense k-point grid. Despite these differences, we did not exclude records based on these parameters, assuming that OQMD, Alexandria, and MP operated within acceptable convergence zones for energy calculations. A similar approach was applied to other VASP parameters, though we welcome feedback on this assumption.366- **Convergence**: Across all databases, we identified numerous records with potentially non-convergent calculations or high-energy configurations, often evidenced by significant atomistic forces. We chose not to exclude these records, as users can filter them easily using the “forces” tag if needed.367- **Energy Above the Hull**: We opted not to filter materials with high energy above the hull, given the current scope of the dataset.368 369The “all” split does not contain any filtering based on this approach, so all records can be downloaded.370 371## De-duplication method and our materials fingerprint372 373For our methods for finding duplicates across databases we creating a hasher function which works the following way:374 375- We compute bonds using the EconNN algorithm already built in Pymatgen376- We create a structure graph from this, encoding the species in the node377- We hash this graph using Weisfeller-Lehman algorithm378- We add symmetry and composition379 380Any structure which has a duplicate based on this method is dropped, only keeping the lowest energy structure. We benchmarked this to be robust to small gaussian noise on atomic positions, lattice vectors, and to respect detected symmetries in a structure. In searching for this method we tried to select one of the more sensitive bonding algorithms that would leave to the least amount of duplicates. We plan on releasing more information on this, as well as code to properly benchmark other fingerprint methods soon.381 382## Check out these helpful spaces to understand the database383 384<figure class="table">385    <table class="ck-table-resized">386        <colgroup>387            <col style="width:50%;">388            <col style="width:50%;">389        </colgroup>390        <tbody>391            <tr>392                <td>393                    <p style="text-align:center;"><figure class="image image_resized"><img src="https://huggingface.co/datasets/LeMaterial/admin/resolve/main/materials_explorer.png"></figure></p>394                </td>395                <td>396                    <p style="text-align:center;"><figure class="image image_resized"><img src="https://huggingface.co/datasets/LeMaterial/admin/resolve/main/Ti_Nb_Sn_LeMat110_PD.png"></figure></p>397                </td>398            </tr>399            <tr>400                <td>401                    <p style="text-align:center;"><a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/spaces/LeMaterial/materials_explorer"><strong>Materials Explorer</strong></a></p>402                </td>403                <td>404                    <p style="text-align:center;"><a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/spaces/LeMaterial/phase_diagram"><strong>Phase Diagram</strong></a></p>405                </td>406            </tr>407            <tr>408                <td>Let's you browse entries in our database, view the crystal structure and its associated properties. Built using <a target="_blank" rel="noopener noreferrer" href="https://github.com/materialsproject/dash-mp-components">Materials Project's dash components</a></td>409                <td>Lets you generate binary and ternary phase diagram using various correction scheme. Built using the Phase Diagram <a target="_blank" rel="noopener noreferrer" href="https://github.com/materialsproject/crystaltoolkit">Crystal Toolkit</a> component by the Materials Project<br><br><u>Disclaimer</u>: the MP2020 correction scheme has not yet been uniformed across datasets, when using this correction scheme please be cautious about interpreting data. We will fix this in upcoming release!</td>410            </tr>411        </tbody>412    </table>413</figure>414 415 416## Stay tuned for future updates417 418We plan to release very soon:419 420- Band gap information on all materials, including direct and indirect band gaps.421- Unification of energy corrections (currently a beta version of this is available for the purpose of the phase diagram application, but please see the disclaimer above).422- Bader charges for all Materials Project materials where possible and the addition of charge data from Alexandria and OQMD423- R2SCAN data from Materials Project424 425In the longer run we plan to release additional datasets including trajectories and surface, adsorbates, and molecules.426 427And more! Stay tuned.428 429## **Support**430 431If you run into any issues regarding feel free to post your questions or comments on any of the following platforms:432 433- [**HF Discussions**](https://huggingface.co/datasets/LeMaterial/LeMat-Bulk/discussions)434- [**Github Issues**](https://github.com/LeMaterial/lematerial/issues)435 436 437## Citation Information438 439We are currently in the process of creating a pre-print to describe our methods, the materials fingerprint method and the dataset. For now however the following can be cited:440 441```442@misc {lematerial_2024,443	author = { {Martin Siron}, {Inel Djafar}, {Lucile Ritchie}, {Etienne Du-Fayet}, {Amandine Rossello}, {Ali Ramlaoui}, {Leandro von Werra}, {Thomas Wolf}, {Alexandre Duval} },444	title = { LeMat-Bulk Dataset },445	year = 2024,446	url = { https://huggingface.co/datasets/LeMaterial/LeMat-Bulk },447	doi = { 10.57967/hf/3762 },448	publisher = { Hugging Face }449}450```451 452[CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) (license used for Materials Project, Alexandria, OQMD) requires proper acknowledgement. 453 454Thus, if you use materials data which include (”mp-”) in the immutable_id, please cite the [Materials Project](https://pubs.aip.org/aip/apm/article/1/1/011002/119685/Commentary-The-Materials-Project-A-materials). 455 456If you use materials data which include (”agm-”) in the immutable_id, please cite [Alexandria, PBE](https://www.science.org/doi/10.1126/sciadv.abi7948) or [Alexandria PBESol, SCAN](https://www.nature.com/articles/s41597-022-01177-w). 457 458If you use materials data which include (”oqmd-”) in the immutable_id, please cite [OQMD](https://link.springer.com/article/10.1007/s11837-013-0755-4). 459 460If you make use of Optimade in your research, please cite [Optimade](https://doi.org/10.1038/s41597-021-00974-z)461 462Finally, if you make use of the Phase Diagram for visualization purposes, or the crystal viewer in the Materials Explorer, please acknowledge [Crystal Toolkit](https://github.com/materialsproject/crystaltoolkit).463 464## License465 466This database is licensed by [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/). 467 468Disclaimer: it is made up of Alexandria, Materials Project and OQMD materials, which are all licensed by [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/). 469