BASF-AI/ChemRxiv-Paragraphs
ChemRxiv Paragraphs This dataset consists of paragraphs from ChemRxiv papers with CC BY 4.0 and CC BY-NC 4.0 licenses, sourced from the BASF-AI/ChemRxiv-Papers dataset. Paragraphs are extracted using Grobid, and filtered using an average log word probability method similar to the approach in allenai/peS2o. Paragraphs with fewer than 50 words are excluded. The number of unique papers in each license category is as follows: CC BY 4.0: 5,848 papers CC BY-NC 4.0: 3,082 papers To… See the full description on the dataset page: https://huggingface.co/datasets/BASF-AI/ChemRxiv-Paragraphs.
ChemRxiv Paragraphs
This dataset consists of paragraphs from ChemRxiv papers with CC BY 4.0 and CC BY-NC 4.0 licenses, sourced from the BASF-AI/ChemRxiv-Papers dataset. Paragraphs are extracted using Grobid, and filtered using an average log word probability method similar to the approach in allenai/peS2o. Paragraphs with fewer than 50 words are excluded.
The number of unique papers in each license category is as follows:
- CC BY 4.0: 5,848 papers
- CC BY-NC 4.0: 3,082 papers
To obtain metadata for each paper, join on the id column with the BASF-AI/ChemRxiv-Papers dataset.
To access paragraphs for a specific license, use the name argument as follows:
import datasets
cc_by = datasets.load_dataset('BASF-AI/ChemRxiv-Paragraphs', name='cc-by')
cc_by_nc = datasets.load_dataset('BASF-AI/ChemRxiv-Paragraphs', name='cc-by-nc')