NP235/AutoMathText

🎉 This work, introducing the AutoMathText dataset and the AutoDS method, has been accepted to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025 Findings)! 🎉

AutoMathText

AutoMathText is an extensive and carefully curated dataset encompassing around 200 GB of mathematical texts. It's a compilation sourced from a diverse range of platforms including various websites, arXiv, and GitHub (OpenWebMath, RedPajama, Algebraic Stack). This rich repository has been autonomously selected (labeled) by the state-of-the-art open-source language model, Qwen-72B. Each piece of content in the dataset is assigned a score `lm_q1q2_score` within the range of [0, 1], reflecting its relevance, quality and educational value in the context of mathematical intelligence.

GitHub homepage: https://github.com/yifanzhang-pro/AutoMathText

ArXiv paper: https://huggingface.co/papers/2402.07625 (https://arxiv.org/abs/2402.07625)

Objective

The primary aim of the AutoMathText dataset is to provide a comprehensive and reliable resource for a wide array of users - from academic researchers and educators to AI practitioners and mathematics enthusiasts. This dataset is particularly geared towards:

• —Facilitating advanced research in the intersection of mathematics and artificial intelligence.
• —Serving as an educational tool for learning and teaching complex mathematical concepts.
• —Providing a foundation for developing and training AI models specialized in processing and understanding mathematical content.

Configs

configs:
  - config_name: web-0.50-to-1.00
    data_files:
      - split: train
        path:
          - data/web/0.95-1.00.jsonl
          - data/web/0.90-0.95.jsonl
          - ...
          - data/web/0.50-0.55.jsonl
    default: true
  - config_name: web-0.60-to-1.00
  - config_name: web-0.70-to-1.00
  - config_name: web-0.80-to-1.00
  - config_name: web-full
    data_files: data/web/*.jsonl
  - config_name: arxiv-0.50-to-1.00
    data_files:
      - split: train
        path:
          - data/arxiv/0.90-1.00/*.jsonl
          - ...
          - data/arxiv/0.50-0.60/*.jsonl
  - config_name: arxiv-0.60-to-1.00
  - config_name: arxiv-0.70-to-1.00
  - config_name: arxiv-0.80-to-1.00
  - config_name: arxiv-full
    data_files: data/arxiv/*/*.jsonl
  - config_name: code-0.50-to-1.00
    data_files:
      - split: train
        path:
          - data/code/*/0.95-1.00.jsonl
          - ...
          - data/code/*/0.50-0.55.jsonl
  - config_name: code-python-0.50-to-1.00
      - split: train
        path:
          - data/code/python/0.95-1.00.jsonl
          - ...
          - data/code/python/0.50-0.55.jsonl
  - config_name: code-python-0.60-to-1.00
  - config_name: code-python-0.70-to-1.00
  - config_name: code-python-0.80-to-1.00
  - config_name: code-jupyter-notebook-0.50-to-1.00
      - split: train
        path:
          - data/code/jupyter-notebook/0.95-1.00.jsonl
          - ...
          - data/code/jupyter-notebook/0.50-0.55.jsonl
  - config_name: code-jupyter-notebook-0.60-to-1.00
  - config_name: code-jupyter-notebook-0.70-to-1.00
  - config_name: code-jupyter-notebook-0.80-to-1.00
  - config_name: code-full
    data_files: data/code/*/*.jsonl

How to load data:

from datasets import load_dataset

ds = load_dataset("math-ai/AutoMathText", "web-0.50-to-1.00") # or any valid config_name

Features

• —Volume: Approximately 200 GB of text data (in natural language and programming language).
• —Content: A diverse collection of mathematical texts, including but not limited to research papers, educational articles, and code documentation.
• —Labeling: Every text is scored by Qwen-72B, a sophisticated language model, ensuring a high standard of relevance and accuracy.
• —Scope: Covers a wide spectrum of mathematical topics, making it suitable for various applications in advanced research and education.

References

• —OpenWebMath [[link]](https://huggingface.co/datasets/open-web-math/open-web-math)
• —RedPajama [[link]](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T)
• —Algebraick Stack [[link]](https://huggingface.co/datasets/EleutherAI/proof-pile-2) (a subset of Proof-Pile-2)

Citation

We appreciate your use of AutoMathText in your work. If you find this repository helpful, please consider citing it and star this repo. Feel free to contact yifanzhangresearch@gmail.com or open an issue if you have any questions (GitHub homepage: https://github.com/yifanzhang-pro/AutoMathText).

@article{zhang2025autonomous,
      title={Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts},
      author={Zhang, Yifan and Luo, Yifan and Yuan, Yang and Yao, Andrew Chi-Chih},
      journal={The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025 Findings)},
      year={2025}
}