dx2102/arxiv-semantic-scholar

📚 arxiv-semantic-scholar

A paper metadata dataset covering every paper on arXiv, with title, authors, full abstract, download link, and submission history.

Each row is additionally enriched by a call to the Semantic Scholar API, adding citation counts and venue information.

🔗 Sources

arXiv metadata: https://www.kaggle.com/datasets/Cornell-University/arxiv (CC0)

Semantic Scholar Academic Graph: https://api.semanticscholar.org/ (ODC-BY)

Snapshot taken 2026-07-04.

📊 Size and columns

3,086,984 rows, one per arXiv paper.

Every arXiv field is represented (cs.CV, cs.LG, hep-ph, quant-ph, math, astro-ph...). Physics, math, and astronomy together make up nearly half.

About 98% of papers have a Semantic Scholar record.

Fields from arXiv: id, title, abstract, authors, authors_parsed, categories, versions, update_date, comments, journal_ref, doi, report_no, license, submitter

Fields nested inside ss_data (JSON) from Semantic Scholar: paperId, externalIds, venue, publicationVenue, publicationDate, citationCount, influentialCitationCount, publicationTypes, fieldsOfStudy, s2FieldsOfStudy, openAccessPdf

🤝 Pairing with Paper Copilot

Paper Copilot (https://github.com/papercopilot/paperlists) provides OpenReview review data, including reviewer scores, decisions, rebuttal metadata, and Google Scholar citation counts (but no full review text).

Some notes:

• —Paper Copilot only covers AI conferences, not the full breadth of arXiv.
• —It mostly contains papers that have already been published or gone through at least one review cycle.
• —For conferences with private review processes (CVPR, ICCV, KDD, etc.), Paper Copilot can only scrape the conference's public paper list, so reviewer scores may not be available.
• —Some papers accepted at AI conferences never appear on arXiv, and Paper Copilot captures those.

There is no shared primary key between this dataset and Paper Copilot. Practical join options:

• —ss_data.externalIds.DBLP against Paper Copilot's dblp field (~65-75% coverage).
• —Exact title match (~51%).

🚀 Loading

from datasets import load_dataset
ds = load_dataset("dx2102/arxiv-semantic-scholar", split="train")

Or download once and convert to a local DuckDB file for fast repeated queries:

from huggingface_hub import hf_hub_download
import duckdb

# Download the parquet to the local HF cache (with progress bar)
path = hf_hub_download(
    repo_id="dx2102/arxiv-semantic-scholar",
    filename="arxiv.parquet",
    repo_type="dataset",
)

# One-time conversion into arxiv.duckdb (a few minutes)
duckdb.connect("arxiv.duckdb").execute(
    f"CREATE TABLE arxiv AS SELECT * FROM '{path}'"
)

# Subsequent queries are fast
con = duckdb.connect("arxiv.duckdb", read_only=True)
con.sql("""
    SELECT title, ss_data->>'citationCount' AS cites
    FROM arxiv
    WHERE lower(title) LIKE '%world model%'
    ORDER BY CAST(cites AS INT) DESC
    LIMIT 20
""").show()