datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
ai-arxiv2-chunksalphabetic-arxiv-authors-it1arxiv-clustering-s2s
ArXivHierarchicalClusteringS2S
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles from arxiv. Clustering of 30 sets, either on the main or secondary category
Task category
t2c
Domains
Academic, Written
Reference
https://www.kaggle.com/Cornell-University/arxiv
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["ArXivHierarchicalClusteringS2S"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/arxiv-clustering-s2s.arxiv_papers
ArXiv Papers
Description
ArXiv is an online open-access repository of over 2.4 million scholarly papers covering fields such as computer science, mathematics, physics, quantitative biology, economics, and more.
When uploading papers, authors can choose from a variety of licenses.
This dataset includes text from all papers uploaded under CC BY, CC BY-SA, and CC0 licenses through a three-step pipeline:
first, the latex source files for openly licensed papers were… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/arxiv_papers.arxiv-clustering-p2p
ArXivHierarchicalClusteringP2P
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles+abstract from arxiv. Clustering of 30 sets, either on the main or secondary category
Task category
t2c
Domains
Academic, Written
Reference
https://www.kaggle.com/Cornell-University/arxiv
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/arxiv-clustering-p2p.arxiv-abstracts-2021
Dataset Card for arxiv-abstracts-2021
Dataset Summary
A dataset of metadata including title and abstract for all arXiv articles up to the end of 2021 (~2 million papers).
Possible applications include trend analysis, paper recommender engines, category prediction, knowledge graph construction and semantic search interfaces.
In contrast to arxiv_dataset, this dataset doesn't include papers submitted to arXiv after 2021 and it doesn't require any external download.… See the full description on the dataset page: https://huggingface.co/datasets/gfissore/arxiv-abstracts-2021.arxiv-abstracts-largeThe arXiv Dataset is a comprehensive knowledge repository of 1.7 million scholarly articles drawn from the vast domains of physics, computer science, statistics, electrical engineering, quantitative biology, and economics among others. It provides open access to vital features such as article titles, authors, categories, abstracts, full text PDFs, and more. The dataset offers immense depth, allowing for exploration into various subdisciplines and interconnections between them. It serves as a… See the full description on the dataset page: https://huggingface.co/datasets/UniverseTBD/arxiv-abstracts-large.arxiv-metadata-2020-2026
arXiv Metadata, enriched (2020–2026)
Per-paper metadata for 1,517,185 arXiv papers spanning 2020-01 → 2026-09,
enriched with abstracts, citation counts, and Semantic Scholar identifiers, and
organized as a two-level hierarchy: field of study → year.
Unlike a bare title index, every record here carries the abstract, the
full author list, citation counts, and the Semantic Scholar corpusId,
so you can do retrieval, classification, citation analysis, and corpus building
directly… See the full description on the dataset page: https://huggingface.co/datasets/yufan/arxiv-metadata-2020-2026.arxiv-tex-corpus-fullarxiv-tex-corpus-full (80GB)
Large-scale LaTeX corpus from arXiv (math, CS, physics, statistics)
📄 Paper: https://arxiv.org/abs/2602.17288
📚 Overview
arxiv-tex-corpus-full (80GB) is a large-scale dataset of LaTeX source content extracted from papers hosted on arXiv.
This version contains approximately 80GB of structured JSONL data, restricted to the following arXiv categories:
math
cs
hep-th
hep-ph
quant-ph
stat.ML
stat.TH
The dataset is designed for research in:
Large… See the full description on the dataset page: https://huggingface.co/datasets/KiteFishAI/arxiv-tex-corpus-full.ai-arxiv
AI ArXiv Dataset
The AI ArXiv dataset contains a selection of papers on the topics of AI and LLMs.
You can find a heavily upgraded v2 dataset here. The v2 dataset improves both data quality and dataset size.
arxiv_abstracts
ArXiv Abstracts
Description
Each paper uploaded to ArXiv includes structured metadata fields, including an abstract summarizing the paper’s findings and contributions.
According to ArXiv’s licensing policy, the metadata for any paper submitted to ArXiv is distributed under the CC0 license, regardless of the license of the paper itself.
Thus, this dataset contains the abstract for every paper submitted to ArXiv through late 2024.
We source the abstracts from ArXiv’s API… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/arxiv_abstracts.ArxivQA
Dataset Card for Mutlimodal Arxiv QA
Dataset Loading Instruction
Each line of the arxivqa.jsonl file is an example:
{"id": "cond-mat-2862",
"image": "images/0805.4509_1.jpg",
"options": ["A) The ordering temperatures for all materials are above the normalized temperature T/Tc T/T_c T/Tc of 1.2.", "B) The magnetic ordering temperatures decrease for Dy, Tb, and Ho as the normalized temperature T/Tc T/T_c T/Tc approaches 1.", "C) The magnetic ordering temperatures for all… See the full description on the dataset page: https://huggingface.co/datasets/MMInstruction/ArxivQA.arxiv_abstracts_filtered
ArXiv Abstracts
Description
Each paper uploaded to ArXiv includes structured metadata fields, including an abstract summarizing the paper’s findings and contributions.
According to ArXiv’s licensing policy, the metadata for any paper submitted to ArXiv is distributed under the CC0 license, regardless of the license of the paper itself.
Thus, this dataset contains the abstract for every paper submitted to ArXiv through late 2024.
We source the abstracts from ArXiv’s API… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/arxiv_abstracts_filtered.arXiv-metadata-oai-snapshot
About Dataset
Dataset name: arXiv academic paper metadata
Data source: https://arxiv.org/
Submission date: 1986-04-25 ~ 2025-05-13 (data updated weekly)
Number of papers: 2,710,806 (as of 2025.5.14)
Fields included: title, author, abstract, journal information, DOI, etc.
Data format: json
Data volume: 4.58G
About ArXiv
For nearly 30 years, ArXiv has served the public and research communities by providing open access to scholarly articles, from the vast branches of… See the full description on the dataset page: https://huggingface.co/datasets/jackkuo/arXiv-metadata-oai-snapshot.mathpile_arxiv_subset
MathPile ArXiv (subset)
Description
This dataset consists of 343,830 TeX files containing mathematics papers sourced from the arXiv. Training and testing sets are already split
Source
The data was obtained from the training + validation portion of the arXiv subset of MathPile.
Format
Given as JSONL files of JSON dicts each containing the single key: "text"
Usage
LaTeX stuff idk
License
The original data is subject to the… See the full description on the dataset page: https://huggingface.co/datasets/aluncstokes/mathpile_arxiv_subset.arxiv-author-affiliations-matched-ror-ids
arXiv Author Affiliations
This dataset contains author affiliation data extracted from arXiv works, matched to Research Organization Registry (ROR) identifiers.
Dataset Description
This dataset was generated from all arXiv works as of 2025/12. The source PDFs were converted to markdown using markitdown, and author affiliations were then extracted using cometadata/affiliation-parsing-lora-Qwen3-8B-distil-GLM_4.5_Air. The extracted affiliations were matched to ROR IDs using… See the full description on the dataset page: https://huggingface.co/datasets/cometadata/arxiv-author-affiliations-matched-ror-ids.raw_arxivArxivMIA
Dataset Card for ArxivMIA
To evaluate various pre-training data detection methods in a more challenging scenario, we introduce ArxivMIA, a new benchmark comprising abstracts from the fields of Computer Science (CS) and Mathematics (Math) sourced from Arxiv.
Repository: https://github.com/zhliu0106/probing-lm-data
Paper: Probing Language Models for Pre-training Data Detection
arxiv-tex-corpus-mediumarxiv-tex-corpus-medium (15GB)
Medium-scale LaTeX corpus from arXiv (math, CS, physics, statistics)
📄 Paper: https://arxiv.org/abs/2602.17288
📚 Overview
arxiv-tex-corpus-medium (15GB) is a medium-sized version of the arXiv LaTeX corpus, containing structured LaTeX source content extracted from selected arXiv categories.
This dataset is restricted to the following categories:
math
cs
hep-th
hep-ph
quant-ph
stat.ML
stat.TH
This version (~15GB) is intended for:
Research… See the full description on the dataset page: https://huggingface.co/datasets/KiteFishAI/arxiv-tex-corpus-medium.metadata_arxivarXiv_Topics
arXiv Topics Dataset
Dataset Summary
The arXiv Topics Dataset provides a structured mapping of arXiv papers to topic categories at three different levels of abstraction. These topic classifications were generated by prompting GPT-4o, ensuring a hierarchical categorization from broad fields to highly specific research areas.
The dataset consists of 2,422,486 paper IDs, each assigned topics across:
Level 1 (Broad Domains): High-level fields such as Computer Science… See the full description on the dataset page: https://huggingface.co/datasets/AliMaatouk/arXiv_Topics.arxiv-rawarena_emb_arxivEmbeddings used for the MTEB Arena.
You can download this repo via
git clone https://hf.co/datasets/mteb/arena_emb_arxiv
cd arena_emb_arxiv
git lfs pull
or just download individual files, e.g. wget https://hf.co/datasets/mteb/arena_emb_arxiv/resolve/main/emb_wikipedia_GritLM__GritLM-7B.json.aa
As there is an upload limit of 50GB per file, we have split files using e.g. split --number=l/6 emb_arxiv_GritLM__GritLM-7B.json emb_arxiv_GritLM__GritLM-7B.json..
To recreate the original file simply… See the full description on the dataset page: https://huggingface.co/datasets/mteb-arena/arena_emb_arxiv.ai-arxiv-chunkedpile_arxivML_arxiv
Dataset Card for 'ML Articles Subset of Scientific Papers' Dataset
Dataset Summary
The dataset consists of 32,621 instances from the 'Scientific papers' dataset, a selection of scientific papers and summaries from ArXiv repository. This subset focuses on articles that are semantically, vocabulary-wise, structurally, and meaningfully closest to articles describing machine learning. This subset was created using sentence embeddings and K-means clustering.
Supported… See the full description on the dataset page: https://huggingface.co/datasets/bakhitovd/ML_arxiv.ArxivBulkDataset
Arxiv Bulk Dataset
This dataset card aims to be a base template for new datasets. It has been generated using this raw template.
Dataset
ArXiv Bulk Dataset
This dataset is a bulk fetch of ArXiv articles, based on the official metadata dataset maintained and updated by Cornell https://www.kaggle.com/datasets/Cornell-University/arxiv/data.
This dataset was created to provide cross-domain academic training data, with existing datasets being domain-specific, and… See the full description on the dataset page: https://huggingface.co/datasets/dankeg/ArxivBulkDataset.arxiv-author-affiliations
Manually Annotated arXiv Preprints Dataset for Structured Extraction of Authors and Affiliations
Dataset Description
This dataset contains manually annotated, structured metadata for a random sample of preprints from arXiv. Each entry in the dataset corresponds to a single publication and includes its title, language, arXiv ID, DOI link, a structured list of authors with their respective affiliations, and the corresponding PDF filename.
Data Fields
Each object… See the full description on the dataset page: https://huggingface.co/datasets/cometadata/arxiv-author-affiliations.ai-arxiv2arxiv-software-repo-links
arXiv Software Repository Links
A dataset mapping arXiv papers (via DOI) to software repositories they reference or which are implementations of the work. Additionaly includes co-citation analysis and community clustering.
Implementation detection is conducted using the evamxb/dev-author-em-clf model from sci-soft-models
Quick Start
from datasets import load_dataset
# Load DOI-to-repo links
links = load_dataset("cometadata/arxiv-software-repo-links", "links")
#… See the full description on the dataset page: https://huggingface.co/datasets/cometadata/arxiv-software-repo-links.