obswork/arxiv-ai-ml-100k-pages

license: other tags:

• —arxiv
• —ocr
• —machine-learning ---

# obswork/arxiv-ai-ml-100k-pages

A page-bounded stratified subset of the raw pool dataset `obswork/arxiv-ai-ml-100k`, filtered to primary subjects cs.AI, cs.CV, cs.LG, and stat.ML.

The raw pool is itself a 100k-paper stratified sample from `Rendra8631/arxiv-papers` at a2c6afb51332d2744b46308df6917697582f8cd4.

## Contents

• —metadata.parquet — 4,866 rows (papers). Includes page_count per paper and the same path columns as the raw pool.
• —pdfs/<primary_code>/<YYMM>/<arxiv_id>.pdf — same layout as the raw pool (YYMM avoids HF's 10k-files-per-directory cap).

## Curation method

• —Per-category page budget: ~25,000 pages (target total ~100,000 pages across four categories).
• —Stratified shuffle with seed 20260418 (independent of the raw pool's paper-level seed 20260416).
• —Papers taken in random order until the running sum of PDF page counts reaches the per-category budget (last paper included in full; slight overshoot allowed).
• —Page counts were measured locally during curation (pypdf, PyMuPDF fallback).

## Realized page totals (post-reconciliation)

• —cs.AI: ~25,015 pages
• —cs.CV: ~25,011 pages
• —cs.LG: ~25,029 pages
• —stat.ML: ~25,001 pages

Total PDF pages (metadata): ~100,056.

## Licensing

Same as the raw pool and upstream arXiv works — see the raw pool README.

## Reproducibility

Pipeline: arxiv_mirror/{curated_filter_plan,curated_mirror,curated_finalize}.py.