Datasetpublic

AlgorithmicResearchGroup/s2orc_arxiv

S2ORC ArXiv A subset of the Semantic Scholar Open Research Corpus (S2ORC) filtered to ArXiv papers. Contains 2.58 million parsed scientific papers with full text, abstracts, structured sections, figures, and citation metadata. Dataset Summary Statistic Value Total papers 2,579,762 Total size ~266 GB Format Parquet Split train Dataset Structure Content Fields Field Type Description title string Paper… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/s2orc_arxiv.

sourceHugging Faceupdated 5mo agoView on Hugging Face

2likes2.7kdownloads

Dataset Card

S2ORC ArXiv

A subset of the Semantic Scholar Open Research Corpus (S2ORC) filtered to ArXiv papers. Contains 2.58 million parsed scientific papers with full text, abstracts, structured sections, figures, and citation metadata.

Dataset Summary

Statistic	Value
Total papers	2,579,762
Total size	~266 GB
Format	Parquet
Split	`train`

Dataset Structure

Content Fields

Field	Type	Description
`title`	string	Paper title
`abstract`	string	Paper abstract
`text`	string	Full paper text
`sections`	list	Structured sections with `title` and `content`
`figures`	list	Figures with `id`, `caption`, and `content`
`references`	list	References with `text` and `doi`
`authors`	list	Author information

Identifier Fields

Field	Type	Description
`corpus_id`	int64	Semantic Scholar corpus ID
`external_ids`	dict	External IDs (`arxiv`, `doi`, `mag`, `acl`, `pubmed`, `dblp`, etc.)

Source & Access Fields

Field	Type	Description
`source`	dict	PDF metadata including `pdf_urls`, `pdf_sha`, and `open_access` info

Usage

python

from datasets import load_dataset

# stream to avoid downloading 266GB
ds = load_dataset("AlgorithmicResearchGroup/s2orc_arxiv", streaming=True, split="train")
for paper in ds:
    print(paper["title"], len(paper["sections"]), "sections")
    break

Related Resources

—S2ORC CS Enriched - CS subset with LLM-generated enrichments
—Algorithmic Research Group - Open Source

Citation

bibtex

@misc{s2orc_arxiv,
    title={S2ORC ArXiv},
    author={Algorithmic Research Group},
    year={2024},
    publisher={Hugging Face},
    url={https://huggingface.co/datasets/AlgorithmicResearchGroup/s2orc_arxiv}
}