rochanaro/hf-arxiv-url-bench

arXiv URL Extraction Benchmark & Longitudinal Corpus

Dataset Description

This repository hosts datasets designed to assess format-specific coverage gaps in URL extraction across the arXiv corpus. The collection facilitates large-scale reproducibility studies and evaluates how different document formats (LaTeX, HTML, XML, Markdown, TXT, PNG) impact automated extraction pipelines.

The repository is divided into two primary corpora:

1. 1.The 200-Paper Benchmark: A curated subset for granular ground-truth analysis.
2. 2.The 360k Longitudinal Corpus: A massive, 33-year (1992–2024) parallel corpus where each paper is successfully represented in six distinct formats (PDF, LaTeX, HTML, XML, Markdown, TXT).

Dataset Sources

1. The 200-Paper Benchmark (Ground Truth)

• —Temporal Coverage: 1992–2024 (Phase A: 2016–24; Phase B: 1992–15)
• —Initial Sampling Pool: 6,090 PDFs stratified by year and document type
• —Final Verified Pool: 200 papers containing all 7 formats (PDF, TEXT, LaTeX, XML, HTML, Markdown, PNG)
• —Total Extracted Data: 2,420 total URLs (2,338 unique URLs) manually audited for strict ground truth

2. The 360k Longitudinal Corpus

• —Temporal Coverage: 1992–2024 (Spanning 33 years)
• —Initial Sampling Pool: 467,467 PDFs (Stratified random sample of up to 15,000 papers per year)
• —Final Common Pool: 364,744 papers (~7,000 - ~13,000 papers per year) containing all 6 formats (PDF, TEXT, LaTeX, XML, HTML, Markdown)

<details> <summary><strong>Repository Structure</strong></summary>

├── .gitattributes
├── README.md
├── arxiv-200-benchmark/
│   ├── arxiv-url-bench-200-raw-files.tar.gz
│   ├── arxiv_extracted_unique_urls_all_200_superset.json
│   ├── arxiv_urls_1992_2024_gt.csv
│   └── intermediate_results/
│       ├── stage_html_urls.json
│       ├── stage_latex_urls.json
│       ├── stage_markdown_urls.json
│       ├── stage_text_urls.json
│       ├── stage_textwal_pdfminer_urls.json
│       ├── stage_textwal_pymupdf_urls.json
│       ├── stage_textwal_pypdf_urls.json
│       ├── stage_textwalcl_urls.json
│       ├── stage_vlm_deepseek_urls.json
│       ├── stage_vlm_minicpm_urls.json
│       ├── stage_vlm_qwen_urls.json
│       └── stage_xml_grobid_urls.json
└── arxiv-mini-corpus/
    ├── arxiv_extracted_urls_5_formats_360k.json
    ├── html.tar.gz
    ├── intermediate_results/
    │   ├── longitudinal_analysis_common_360k_arxiv_ids.json
    │   ├── stage1_sampled_ids.json
    │   ├── stage2_pdf_textwal_urls.json
    │   ├── stage3_latex_urls.json
    │   ├── stage3b_latex_sources_all.json
    │   ├── stage4_grobid_xml_master_summary.csv
    │   ├── stage4_xml_grobid_urls.json
    │   ├── stage5_markdown_marker_success.json
    │   ├── stage5_markdown_urls.json
    │   ├── stage6_html_urls.json
    │   └── stage6_latexml_success.jsonl
    ├── latex.tar.gz
    ├── markdown.tar.gz
    ├── pdf.tar.gz
    ├── textwal.tar.gz
    └── xml.tar.gz

</details>

Dataset Structure & Storage

Warning: The full longitudinal corpus (~360k) is exceptionally large. Ensure you have adequate storage and bandwidth before executing a full download.

1. The 200-Paper Benchmark Corpus

File: `arxiv-200-benchmark/arxiv-url-bench-200-raw-files.tar.gz`

• —Size: ~2.8 GB
• —Description: Contains the raw files for the 200 benchmark papers across their different file format variants.

2. The 200-Paper Manual Annotations & OADS Categories

File: `arxiv-200-benchmark/arxiv_urls_1992_2024_gt.csv`

• —Description: Contains all manual ground-truth annotations for the 200-paper benchmark, featuring explicit OADS (Open Access Data and Software) category labels along with structural and contextual complexity evaluations for each URL.

3. The 200-Paper Unique URLs Superset (including Ground Truth)

File: `arxiv-200-benchmark/arxiv_extracted_unique_urls_all_200_superset.json`

• —Size: ~10 MB
• —Description: Registry of all extracted URLs across all format variants, including the human-annotated ground truth labels.

<details> <summary><strong>📋 Show JSON schema for <code>arxivextracteduniqueurlsall200superset.json</code></strong></summary>

Top-level keys are arXiv IDs. Each entry contains one sub-object per extraction source (formats variants, VLM outputs, and the merged superset):

{
  "arxiv_id": {
    "text": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "html": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "latex": {
      "url_count": "int",
      "urls": ["str"]
    },
    "xml": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "vlm_qwen": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "vlm_deepseek": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "vlm_minicpm": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "textwal": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "textwal_pypdf": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "textwal_pdfminer": {
      "url_count": "int",
      "filename": "str",
      "urls": ["str"]
    },
    "textwalcl": {
      "urls": ["str"],
      "number_of_pages": "int",
      "input_tokens": "int",
      "output_tokens": "int",
      "cost_usd": "float",
      "runtime_sec": "float",
      "failed": "bool"
    },
    "markdown": {
      "filename": "str",
      "md_path": "str",
      "url_count": "int",
      "urls": ["str"]
    },
    "pdf": {
      "urls": ["str"],
      "url_count": "int"
    },
    "oads": {
      "oads_info": ["str"],
      "url_count": "int"
    },
    "superset": {
      "urls": ["str"],
      "url_count": "int"
    }
  }
}

Field notes:

• —text, html, xml, latex, markdown, pdf — per-format extraction results, each with the URLs found (urls), the count (url_count), and, where applicable, the source filename/md_path.
• —vlm_qwen, vlm_deepseek, vlm_minicpm — extraction results from the respective vision-language models run over the paper's page images.
• —textwal, textwal_pypdf, textwal_pdfminer — text-based extraction variants using different underlying parsing libraries.
• —textwalcl — Claude-based extraction (token usage, cost, runtime, failure status) in addition to the URLs found.
• —oads — Open Access Data and Software category info for the URLs in this entry.
• —superset — the deduplicated union of URLs across all extraction sources for this paper.

</details>

3b. The 200-Paper Intermediate, Per-Stage Extraction Results

Folder: `arxiv-200-benchmark/intermediate_results/`

• —Description: Per-stage/per-method URL extraction outputs for the 200-paper benchmark, prior to being merged into the unique URLs superset (item 3, above). Each file corresponds to a single extraction source:
• —stage_html_urls.json, stage_latex_urls.json, stage_markdown_urls.json, stage_text_urls.json, stage_xml_grobid_urls.json — per-format extraction results.
• —stage_textwal_pypdf_urls.json, stage_textwal_pdfminer_urls.json, stage_textwal_pymupdf_urls.json — text-based extraction variants using different underlying PDF-parsing libraries.
• —stage_textwalcl_urls.json — Claude-based extraction results.
• —stage_vlm_qwen_urls.json, stage_vlm_deepseek_urls.json, stage_vlm_minicpm_urls.json — extraction results from the respective vision-language models run over the paper's page images.

4. The 360k Longitudinal Multi-Format Corpus

Files: `pdf.tar.gz,latex.tar.gz,html.tar.gz,textwal.tar.gz,xml.tar.gz,markdown.tar.gz,`

• —Size: ~560 GB
• —Description: The complete, multi-format synchronized common pool containing 364,744 papers.

5. The 360k Longitudinal Extracted URLs

File: `arxiv_extracted_urls_5_formats_360k.json`

• —Size: ~690 MB
• —Description: Compiled output of every URL extracted from the 360k dataset across the 5 core text formats.

<details> <summary><strong>📋 Show JSON schema for <code>arxivextractedurls5formats_360k.json</code></strong></summary>

Top-level keys are publication years, each mapping to arXiv IDs published that year. Each arXiv ID contains one sub-object per format, with the extracted URLs and a count:

{
  "year": {
    "arxiv_id": {
      "textwal": {
        "urls": ["str"],
        "num_urls": "int"
      },
      "latex": {
        "urls": ["str"],
        "num_urls": "int"
      },
      "xml": {
        "urls": ["str"],
        "num_urls": "int"
      },
      "markdown": {
        "urls": ["str"],
        "num_urls": "int"
      },
      "html": {
        "urls": ["str"],
        "num_urls": "int"
      }
    }
  }
}

Field notes:

• —year — four-digit publication year (e.g. "2016"), used to shard the 364,744 papers.
• —arxiv_id — arXiv identifier of the paper.
• —textwal, latex, xml, markdown, html — one sub-object per format, each holding the list of extracted urls and the corresponding num_urls count.

</details>

6. The 360k Longitudinal Corpus Intermediate, Per-Stage Pipeline Files

Folder: `arxiv-mini-corpus/intermediate_results/`

• —Description: Per-stage outputs from the pipeline used to build the 360k longitudinal corpus, prior to compilation into the 5-formats extraction file (item 5, above):
• —stage1_sampled_ids.json — the initial stratified sample of arXiv IDs.
• —stage2_pdf_textwal_urls.json — URLs extracted from PDF text.
• —stage3_latex_urls.json / stage3b_latex_sources_all.json — LaTeX-derived URLs and the full set of retrieved LaTeX sources.
• —stage4_xml_grobid_urls.json / stage4_grobid_xml_master_summary.csv — URLs extracted from GROBID-generated XML, with a master run summary.
• —stage5_markdown_urls.json / stage5_markdown_marker_success.json — Markdown-derived URLs and Marker conversion success log.
• —stage6_html_urls.json / stage6_latexml_success.jsonl — HTML-derived URLs and LaTeXML conversion success log.
• —longitudinal_analysis_common_360k_arxiv_ids.json — the final common pool of 364,744 arXiv IDs present across all six formats.