marin-community/ar5iv-no-problem-markdown

Marin Markdownified Ar5iv

Markdownified Ar5iv transforms academic papers from arXiv into clean, structured Markdown format consisting of 2.74B tokens across two splits. This dataset preserves th content while making it accessible for language model training on academic text.

Processing and Cleaning Pipeline

Our conversion pipeline combines several sophisticated techniques to transform raw Wikipedia HTML into high-quality Markdown:

1. 1.HTML Preprocessing: We start with the Ar5iv dump in Extended DOLMA format, which provides HTML representations of academic papers with metadata.

1. 1.Structural Cleanup
2. 2.The abstract is transformed into a proper section heading for consistent document structure
3. 3.LaTeX equations are carefully preserved using inline ($...$) and display ($$...$$) notation
4. 4.Code blocks and listings maintain proper formatting with appropriate line breaks

1. 1.Noise Reduction:
2. 2.Author information is removed
3. 3.Title page elements are streamlined to avoid redundancy
4. 4.The Ar5iv footer is removed to eliminate conversion metadata
5. 5.Figure captions are removed to focus on the main content
6. 6.Bibliography sections, footnotes, and citation links are removed

1. 1.Formatting Cleanup:
2. 2.List items are cleaned to prevent duplicate numbering patterns (e.g., "1. 1.")
3. 3.Content before the first main section (typically metadata) is removed
4. 4.Equation tables are converted to inline elements for better rendering

1. 1.DOM Simplification: We employ a custom-enhanced version of Resiliparse that preserves semantic HTML structure. Rather than flattening to plain text, we retain important elements like headings, paragraphs, lists while removing scripts, tracking code, and boilerplate.

1. 1.Markdown Conversion: Our custom Markdownify implementation transforms the simplified DOM into clean Markdown. The final output stores each article as a JSON object containing the Markdown text and essential metadata.

Dataset Variants

The Markdownified Ar5iv dataset comes in two variants:

1. 1.Ar5iv No Problem (2.74B tokens): Papers that were converted without significant issues or warnings during the HTML generation process. This subset represents the cleanest and most reliable papers.
2. 2.Ar5iv Warning (19.6B tokens): Papers that generated warnings during conversion from LaTeX to HTML. While still valuable, these may contain occasional formatting artifacts.

Usage Example

from datasets import load_dataset

ds = load_dataset(
    "marin-community/ar5iv-no-problem-markdown",
    split="train",
    streaming=True
)

for article in ds.take(3):
    print(article["text"])

Citation

If you use this dataset in your research, please cite both the original Wikipedia contributors and our work:

@misc{markdownified_ar5iv_2024,
  title        = {Markdownified Ar5iv},
  author       = {The Marin Community},
  year         = {2024},
  url          = {https://huggingface.co/datasets/marin-community/ar5iv-no-problem-markdown}
}

License

All content inherits Ar5iv's licensing: C-UDA-1.0. Our conversion tools and pipeline are released under Apache 2.0.

Acknowledgement

We extend our gratitude to:

• —Arxiv Labs and KWARC for their work on the Ar5iv dataset
• —Janek Bevendorff for the Resiliparse project
• —Matthew Dapena-Tretter for Markdownify