AlgorithmicResearchGroup/s2orc_full

S2ORC Full — Semantic Scholar Open Research Corpus

A complete redistribution of the S2ORC dataset in Parquet format on Hugging Face, containing 14.5 million academic papers with full text, structured metadata, and citation information.

Dataset Description

S2ORC (Semantic Scholar Open Research Corpus) is a general-purpose corpus for NLP and text mining research over scientific papers, originally developed by the Allen Institute for AI. This version provides the full dataset in a convenient, ready-to-use format via the Hugging Face Hub.

The corpus spans all major academic disciplines — computer science, biomedicine, physics, mathematics, social sciences, and more — making it suitable for large-scale scientific language modeling, information extraction, citation analysis, and document understanding tasks.

Schema

Quick Start

Stream (no full download required)

from datasets import load_dataset

ds = load_dataset("AlgorithmicResearchGroup/s2orc_full", split="train", streaming=True)

for paper in ds:
    print(paper["title"])
    print(paper["abstract"][:200])
    break

Load with Hugging Face Datasets

from datasets import load_dataset

# Full dataset (~1.4 TB) — make sure you have enough disk space
ds = load_dataset("AlgorithmicResearchGroup/s2orc_full", split="train")

Load with Dask (for distributed processing)

import dask.dataframe as dd

df = dd.read_parquet("hf://datasets/AlgorithmicResearchGroup/s2orc_full/data/")

Load with Polars

import polars as pl

df = pl.read_parquet("hf://datasets/AlgorithmicResearchGroup/s2orc_full/data/train-00000-of-02690.parquet")

Use Cases

• —Scientific language modeling — Pre-train or fine-tune language models on academic text
• —Citation analysis — Study citation networks and reference patterns across disciplines
• —Information extraction — Extract entities, relations, and structured knowledge from papers
• —Document summarization — Train abstractive or extractive summarization models on paper/abstract pairs
• —Bibliometric research — Analyze publication trends, authorship patterns, and cross-disciplinary connections
• —Retrieval-augmented generation — Build retrieval systems over the scientific literature

Citation

If you use this dataset, please cite the original S2ORC paper:

@inproceedings{lo-etal-2020-s2orc,
    title = "{S2ORC}: The Semantic Scholar Open Research Corpus",
    author = "Lo, Kyle and Wang, Lucy Lu and Neumann, Mark and Kinney, Rodney and Weld, Daniel",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.acl-main.447",
    doi = "10.18653/v1/2020.acl-main.447",
    pages = "4969--4983"
}

License

This dataset is released under the Open Data Commons Attribution License (ODC-By 1.0). You are free to share and adapt the data, provided you give appropriate attribution.