Datasetpublic

naveenmarthala/arxiv-latex

arXiv LaTeX Source Dataset This dataset provides the entire corpus of arXiv's LaTeX source files, pre-parsed, formatted, and aligned with official metadata in ready-to-query Parquet files. Why I Built This If you have ever tried to work with the complete history of arXiv papers at scale, you have likely run into two massive hurdles: Network Egress Costs: While arXiv does offer public bulk access to its source files via S3 (s3://arxiv), the bucket is configured… See the full description on the dataset page: https://huggingface.co/datasets/naveenmarthala/arxiv-latex.

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes653downloads

Dataset Card

arXiv LaTeX Source Dataset

This dataset provides the entire corpus of arXiv's LaTeX source files, pre-parsed, formatted, and aligned with official metadata in ready-to-query Parquet files.

Why I Built This

If you have ever tried to work with the complete history of arXiv papers at scale, you have likely run into two massive hurdles:

1.Network Egress Costs: While arXiv does offer public bulk access to its source files via S3 (s3://arxiv), the bucket is configured as "requester-pays." If you attempt to download the full 5 TB corpus of over 3 million papers to any machine outside of the AWS us-east-1 (N. Virginia) region, you are hit with standard AWS egress fees. At $0.09 per GB, a single full download costs more than $450.
2.Computational Friction: The raw S3 data is packaged as hundreds of nested .tar archives, each containing gzip payloads (.gz) of individual papers. Extracting these, parsing the inner LaTeX code, and matching the files with their JSON metadata snapshots is extremely CPU-heavy, requiring complex local pipeline architecture.

This dataset acts as an open mirror that solves both issues. The project ingests the S3 data inside us-east-1 (where data transfer is free), index and process the LaTeX source documents, align them directly to their metadata snapshot, and upload the finalized Parquet files here. Researchers and developers can download clean, structured data without worrying about network egress bills or spending days writing ingestion code.

Ingest Schedule & The Parquet Manifest

—Update Cycle: I sync the latest publications and revisions from arXiv S3 once every month.
—Manifest Tracking: To support crash-resilient resuming, validation, and incremental syncing, the project maintains a central XML manifest file: arxiv_parquet_manifest.xml. This manifest maps each Parquet partition file to its size, MD5 checksum, processed timestamp, range of paper IDs (first_item and last_item), and the list of raw S3 .tar files that were unpacked to generate it.

Dataset Schema

Every row represents a single paper with metadata and parsed LaTeX source contents:

Column Name	Type	Description
`id`	`string`	arXiv paper identifier (e.g. `0704.0001` or `hep-th/9901001`).
`yymm_id`	`string`	Normalized ID mapped to `YYMM` format for chronological sorting.
`submitter`	`string`	Name of the user who uploaded the paper.
`authors`	`string`	Raw authors string.
`title`	`string`	Title of the paper.
`comments`	`string`	Submitter comments or journal references.
`journal-ref`	`string`	Official journal publication reference (if published).
`doi`	`string`	Digital Object Identifier (DOI).
`report-no`	`string`	Report or document series numbers.
`categories`	`string`	Space-separated arXiv categories (e.g., `cs.CL math.PR`).
`license`	`string`	License under which the paper was published.
`abstract`	`string`	The paper's abstract.
`versions`	`list<struct>`	Struct list of versions with creation timestamps.
`update_date`	`string`	Date the paper record was last modified by arXiv.
`authors_parsed`	`list<list<string>>`	Split author names (structured by Last Name, First Name, suffix).
`latex`	`large_string`	The parsed, compiled LaTeX source code from the paper. All source files (`.tex`, `.bib`, `.sty`, etc.) are bundled into a single readable Markdown-style tree structure.

Curation & Licensing

This dataset mirrors data provided under arXiv's Terms of Use. The copyright and licenses of individual paper contents are retained by their respective authors, and correspond to the license identifier specified in the license column.