nielsr/arxiv-chandra-ocr-250-20260401-l40sx1

arXiv OCR with Chandra OCR 2

This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.

Summary

• —Output dataset: nielsr/arxiv-chandra-ocr-250-20260401-l40sx1
• —Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-250-20260401-l40sx1
• —Source paper IDs in input list: 27,584
• —Processed IDs recorded in state/processed_ids.txt: 250
• —Successes: 250
• —Partial successes: 0
• —Errors: 0
• —Next shard index: 25
• —Updated at: 2026-04-01T16:24:48.639101+00:00

Files

• —data/part-*.jsonl.gz: OCR result shards, one JSON object per paper
• —state/processed_ids.txt: completed paper IDs used for resume
• —state/summary.json: aggregate counters and bookkeeping

Each paper record includes:

• —num_pages: total number of pages in the source PDF
• —num_pages_processed: number of pages actually sent to OCR
• —pdf_exceeds_page_limit: whether the PDF had more pages than the configured OCR cap
• —max_pages_per_paper: configured OCR page cap for the run

Load the results

from datasets import load_dataset

dataset = load_dataset("nielsr/arxiv-chandra-ocr-250-20260401-l40sx1", data_files="data/*.jsonl.gz", split="train")
print(dataset[0]["paper_id"])
print(dataset[0]["markdown"][:1000])

Job config

• —Prompt type: ocr_layout
• —Page batch size: 28
• —Max output tokens: 12384
• —Max model length: 18000
• —GPU memory utilization: 0.85
• —Minimum arXiv request interval: 3.1 seconds
• —Max pages per paper sent to OCR: 30
• —Bucket backend: hf-cli
• —Paginate output: False
• —Include headers/footers: False

Reproduction

hf jobs uv run --flavor l4x1 --image vllm/vllm-openai:v0.17.0 \
  -s HF_TOKEN --timeout 2d \
  ./chandra2-arxiv-ocr.py --output-dataset nielsr/arxiv-chandra-ocr-250-20260401-l40sx1 \
  --output-bucket nielsr/arxiv-chandra-ocr-250-20260401-l40sx1 \
  --paper-ids-url https://.../hf_missing_paper_ids.txt