huawei-csl/GitScholar

GitScholar: arXiv AI papers, their citations, and their GitHub footprint

GitScholar is a relational, fully timestamped dataset linking AI arXiv papers to their citation history on Semantic Scholar and to the GitHub repositories that reference them. It is built for studying, and predicting, how research papers gain attention over time: every row carries the date on which it became true, so the state of the whole graph can be reconstructed as of any day between 1991 and 2026-06-16.

Contents

The dataset is twelve Parquet files in two groups that share one key, the arXiv id.

Academic side, from arXiv metadata and the Semantic Scholar (S2) bulk dump:

GitHub side, from a crawl of every public repository whose README mentions arXiv:

Scope

Papers. A paper is in scope if it is on arXiv with at least one of the categories cs.AI, cs.LG, cs.CV or cs.CL, whether as primary or cross-listed category, and if Semantic Scholar holds a record for it. Papers are dated by the earlier of their first arXiv submission and their S2 publication date.

Authors. Every S2 author of a paper in scope. The author neighbourhood tables (edges_author_paper, events_citation_paper) follow these authors across their entire publication record, arXiv or not and in any field, so that an author's standing at a given date can be computed. They are supersets of the two arXiv tables: semi-joining either on features_arxiv.corpusid recovers the in-scope view.

Repositories. Every public GitHub repository whose README contained the string "arxiv" when the crawl found it, created between 2008 and the freeze. The crawl runs GitHub's search API over creation-date windows, so it is exhaustive within GitHub's own indexing of READMEs at crawl time.

Cutoff. Every table ends on 2026-06-16. Nothing dated later appears anywhere, so a point-in-time query as of any earlier day sees only what had been observed by then.

Schemas

features_arxiv

Titles, abstracts and categories are not included here; they can be found in the public arXiv metadata snapshot https://www.kaggle.com/datasets/Cornell-University/arxiv and join on id.

edges_author_arxiv, edges_author_paper

In edges_author_paper, papers with no S2 publication date are dated by their earliest fully dated citation. Papers with no dated citation are omitted.

edges_citation_arxiv

The citation graph restricted to papers in scope: both ends are in features_arxiv, so it can be used directly as paper-to-paper structure. It covers 507,325 citing and 379,972 cited papers.

The edge is dated by the citing paper's date as features_arxiv carries it, so that a paper has one date throughout the dataset. The events tables below date citations by the citing paper's S2 publication date instead; the two agree for 99.9 percent of these edges and differ only where arXiv posted the citing paper before S2's date. Every edge has a date, since every citing paper is in scope. Self-citations of a record by itself, an S2 merge artifact, are removed.

events_citation_arxiv, events_citation_paper

A citation is dated by the publication date of the citing paper, since the citation edges themselves carry no date. Every citation in the S2 dump into a paper of the table is represented exactly once, in one of five buckets:

Nothing is dropped. A consumer who wants ordinary citations filters on bucket == "post" and can still see what that excluded. A consumer who wants the year-only citations on a timeline chooses their own rule for placing them: a uniform random day within the year, the year's midpoint, or the paper's own date when the years coincide.

features_repo

edges_repo_arxiv

Links are recovered from README text: arxiv.org URLs in all their forms, arXiv:NNNN.NNNNN inline references, and eprint fields of BibTeX entries. The README is read from the repository's commit history, so a link is dated by the commit that introduced or removed it, collapsed to one state per day. Three consequences:

• —The link set is a change log. The links a repository has on day D are the pairs whose latest event on or before D is an add. Roughly 30 percent of rows are removals; paper feed repositories in particular rotate their links daily.

arxiv_id is not restricted to papers in features_arxiv. Every arXiv link found is kept, in any field, so the table can be joined to any arXiv-keyed dataset. Within this release, join on features_arxiv.id to restrict to the in-scope papers.

events_stars, events_forks, events_issues, events_prs

Counts are daily gains, not running totals, and days with no gain have no row. Stars come from GitHub's per-star starredAt stream and were never decremented for unstars, so a repository's summed stars can slightly exceed its displayed count.

Quality filters applied to the paper set

These remove papers whose metadata is wrong in a way that would corrupt every date-based quantity derived from them.

• —Author-count disagreement. Papers where arXiv and S2 disagree by more than two authors are dropped: one of the two records is typically a different paper or a corrupted merge. This is not a common case.
• —Late arXiv postings. Papers with at least 25 citations, of which at least a quarter predate the paper's own date, are dropped. This catches well cited papers that were published in conferences, then uploaded to arXiv, and S2 attributes the arXiv date instead of the conference one.
• —Papers with no usable author id are dropped, since they would carry an empty author record that disagrees with n_authors.
• —Papers dated before 1991-08-14, the day arXiv opened, are dropped as impossible.

Papers excluded by the first two filters are also kept out of the author neighbourhood, so that they cannot re-enter through their co-authors.

Duplicate S2 records for one arXiv id, which happen when a preprint and its published version are never merged, are resolved to the lowest corpus id: the record created first and the one citations accumulate against.

Known limitations

• —Citation timing is reconstructed from the citing paper's date, so a citation from a paper S2 dates to a journal issue can appear months after the work was actually circulating, and S2's own dating errors propagate.
• —The crawl finds repositories through GitHub's README search, which indexes only the default branch's README
• —Link detection is textual. A repository mentioning a paper in a reading list and one implementing it produce the same edge. Repositories that link many papers can be identified from the edge table and treated separately.
• —Author identity is Semantic Scholar's author disambiguation, with its known splitting and merging errors.

Sources, licensing and availability

GitScholar is available at https://huggingface.co/datasets/MrGuanda/GitScholar and is released under the Open Data Commons Attribution License (ODC-By), which allows reuse and modification with appropriate attribution.

It combines:

• —arXiv paper metadata, obtained through the arXiv Open Archives Initiative (OAI) interface. Only identifiers and dates are redistributed here; titles, abstracts and categories remain in the public arXiv metadata and join on the arXiv id.
• —Semantic Scholar paper, author and citation records from the Semantic Scholar Open Research Corpus (Kinney et al., 2023), bulk release of 2026-06-16.
• —GitHub public repository metadata, collected through the official GitHub GraphQL API in accordance with GitHub's API Terms of Use. Only public metadata is included: repository names, creation dates, the arXiv identifiers found in READMEs, and daily counts of stars, forks, issues and pull requests. No personal data, no user-generated content such as README text, issue text or commit messages, and no private repository information is included.

Please cite the underlying sources alongside GitScholar when using it.

Reading the data

All files are standard Parquet and open with any Parquet reader. In Python:

import polars as pl

papers = pl.read_parquet("features_arxiv.parquet")
events = pl.read_parquet("events_citation_arxiv.parquet")

# Citations at one year after publication, ordinary citations only
cit_1y = (
    events.filter(pl.col("bucket") == "post")
    .join(papers.select("corpusid", pl.col("date").alias("pub_date")), on="corpusid")
    .filter(pl.col("date") <= pl.col("pub_date").dt.offset_by("1y"))
    .group_by("corpusid").agg(pl.col("citations").sum())
)

# Stars of every repository linking a paper, as of a cutoff date
cutoff = pl.date(2025, 6, 17)
links = (
    pl.read_parquet("edges_repo_arxiv.parquet")
    .filter(pl.col("date") <= cutoff).sort("date")
    .group_by("repo_id", "arxiv_id").agg(pl.col("added").last())
    .filter("added")
)
stars = (
    pl.read_parquet("events_stars.parquet")
    .filter(pl.col("date") <= cutoff)
    .group_by("repo_id").agg(pl.col("count").sum().alias("stars"))
)
stars_per_paper = links.join(stars, on="repo_id", how="left").group_by("arxiv_id").agg(pl.col("stars").sum())