obswork/arxiv-ai-ml-100k-papers

license: other tags:

• —arxiv
• —ocr
• —machine-learning ---

# obswork/arxiv-ai-ml-100k

A 99,999-paper stratified subset of `Rendra8631/arxiv-papers` at revision a2c6afb51332d2744b46308df6917697582f8cd4, filtered to the primary subjects cs.AI, cs.CV, cs.LG, and stat.ML. Only papers submitted in 2023-2025 are included.

This dataset is a build artifact of the OCR benchmark in modal-model-experiments/arxiv_mirror/. It exists so that the downstream benchmark corpus (rasterized page images) is built from a stable, well-scoped PDF pool.

## Contents

• —metadata.parquet - 99999 rows, one per paper. Carries the source schema plus derived primary_code, submission_year, and target_filename columns.
• —pdfs/<primary_code>/<YYMM>/<arxiv_id>.pdf - one file per paper, bucketed by primary arXiv subject code (e.g. cs.CV) and submission year-month. The YYMM layer keeps every directory under the HF 10k-files-per-dir cap and gives consumers a natural temporal slice.

## Filter and sampling

• —Primary subjects kept: cs.AI, cs.CV, cs.LG, stat.ML.
• —Submission year kept: >= 2023 (derived from the arxivid `YYMM.NNNNN` prefix, falling back to a regex over `submissiondate` for any pre-2007 old-format IDs).
• —Version dedup: for papers with multiple versions in the source metadata (...v1, ...v2), only the latest version is kept.
• —Stratified random sample of 25000 papers per category, numpy.random.default_rng(seed=20260416). Categories with fewer than 25000 available rows contribute all of their rows; any resulting shortfall is topped up uniformly from the remaining filtered pool.
• —Per-category final counts:

• —cs.AI: 11788 papers
• —cs.CV: 43616 papers
• —cs.LG: 39599 papers
• —stat.ML: 4996 papers

## Licensing

This dataset is a collection of individual scholarly works from arXiv.org. Each paper remains under the copyright and license chosen by its respective author(s) (typically Creative Commons). Users of this dataset are responsible for checking the specific license terms of any paper they download or redistribute. See the source dataset for upstream licensing notes.

## Reproducibility

Source commit: a2c6afb51332d2744b46308df6917697582f8cd4. Build pipeline: arxiv_mirror/{filter_plan,mirror,finalize}.py.