datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
arxiv-cs-2020-2025-pdfsarxiv-papers-by-subject
arXiv Papers by Subject
A reorganised version of the nick007x/arxiv-papers dataset, partitioned by subject code, year, and month for efficient selective access.
Dataset Description
This dataset contains metadata for over 2.5 million arXiv papers, organised into a hierarchical directory structure that allows users to download only the specific subjects and time periods they need, rather than the entire dataset.
Motivation
The original… See the full description on the dataset page: https://huggingface.co/datasets/permutans/arxiv-papers-by-subject.arxiv-complete
arXiv Complete Corpus
A snapshot of arXiv's metadata, version history, submission files and rendered
documents. It covers 3,148,796 papers and includes file contents, paths, sizes
and SHA-256 digests. Metadata comes from arXiv's OAI-PMH arXivRaw interface;
files come from the GCS mirror, S3 source archives and direct PDF fetches.
This release holds a PDF for 99.47% of papers and 99.54% of versions reported
with a non-zero submission size. It is a one-off snapshot; coverage gaps… See the full description on the dataset page: https://huggingface.co/datasets/secemp9/arxiv-complete.arxiv-latex
arXiv LaTeX Source Dataset
This dataset provides the entire corpus of arXiv's LaTeX source files, pre-parsed, formatted, and aligned with official metadata in ready-to-query Parquet files.
Why I Built This
If you have ever tried to work with the complete history of arXiv papers at scale, you have likely run into two massive hurdles:
Network Egress Costs: While arXiv does offer public bulk access to its source files via S3 (s3://arxiv)… See the full description on the dataset page: https://huggingface.co/datasets/scholarweave/arxiv-latex.ArxivCap
Dataset Card for ArxivCap
Data Instances
Example-1 of single (image, caption) pairs
"......" stands for omitted parts.
{
'src': 'arXiv_src_2112_060/2112.08947',
'meta':
{
'meta_from_kaggle':
{
'journey': '',
'license': 'http://arxiv.org/licenses/nonexclusive-distrib/1.0/',
'categories': 'cs.ET'
},
'meta_from_s2':
{
'citationCount': 8… See the full description on the dataset page: https://huggingface.co/datasets/MMInstruction/ArxivCap.arxiv-ai-ml-100k-papers
license: other
tags:
- arxiv
- ocr
- machine-learning
---
# obswork/arxiv-ai-ml-100k
A 99,999-paper stratified subset of
[`Rendra8631/arxiv-papers`](https://huggingface.co/datasets/Rendra8631/arxiv-papers)
at revision `a2c6afb51332d2744b46308df6917697582f8cd4`, filtered to the primary
subjects `cs.AI`, `cs.CV`, `cs.LG`, and `stat.ML`. Only papers submitted in `2023`-`2025` are included.
This dataset is a build artifact of the OCR… See the full description on the dataset page: https://huggingface.co/datasets/obswork/arxiv-ai-ml-100k-papers.Arxiv_2025_OCR
Arxiv_2025_OCR
OCR Data.
arxiv_dbArxiv_2024_OCR
Arxiv_2024_OCR
OCR Data.
arxiv-summarization
Arxiv dataset for summarization
Dataset for summarization of long documents.Adapted from this repo.Note that original data are pre-tokenized so this dataset returns " ".join(text) and add "\n" for paragraphs. This dataset is compatible with the run_summarization.py script from Transformers if you add this line to the summarization_name_mapping variable:
"ccdv/arxiv-summarization": ("article", "abstract")
Data Fields
id: paper id
article: a string containing the body of… See the full description on the dataset page: https://huggingface.co/datasets/ccdv/arxiv-summarization.ai-arxiv2-chunksarxiv_qa2MINT-1T-ArXiv
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-ArXiv.Arxiv_2023_OCR
Arxiv_2023_OCR
OCR Data.
Arxiv_2021_OCR
Arxiv_2021_OCR
OCR Data.
Arxiv_2020_OCR
Arxiv_2020_OCR
OCR Data.
arxiv-latex-5TThe dataset used for https://github.com/TIGER-AI-Lab/ScholarCopilot.
arxiv-ai-ml-100k-pages
license: other
tags:
- arxiv
- ocr
- machine-learning
---
# obswork/arxiv-ai-ml-100k-pages
A **page-bounded** stratified subset of the raw pool dataset
[`obswork/arxiv-ai-ml-100k`](https://huggingface.co/datasets/obswork/arxiv-ai-ml-100k),
filtered to primary subjects `cs.AI`, `cs.CV`, `cs.LG`, and `stat.ML`.
The raw pool is itself a 100k-paper stratified sample from… See the full description on the dataset page: https://huggingface.co/datasets/obswork/arxiv-ai-ml-100k-pages.arxiv-papers1-sentence-level-gutenberg-en_arxiv_pubmed_sodaalphabetic-arxiv-authors-it1arxiv-metadata-snapshot
Dataset Card for "arxiv-metadata-oai-snapshot"
More Information needed
This is a mirror of the metadata portion of the arXiv dataset.
The sync will take place weekly so may fall behind the original datasets slightly if there are more regular updates to the source dataset.
Metadata
This dataset is a mirror of the original ArXiv data. This dataset contains an entry for each paper, containing:
id: ArXiv ID (can be used to access the paper, see below)
submitter:… See the full description on the dataset page: https://huggingface.co/datasets/librarian-bots/arxiv-metadata-snapshot.arx-left-cube
ARX Left Cube YuHai HDF5 Dataset
This dataset contains ARX-X5 left-arm single-arm teleoperation episodes for a cube manipulation task.
Each episode is stored as one HDF5 file.
https://huggingface.co/datasets/Xia-2004/arx-left-cube
Files
episode_000000.hdf5 ... episode_000200.hdf5
201 episodes
63,619 total frames
RGB images are stored as uint8
Actions are stored as float32
HDF5 Format
Each episode_*.hdf5 contains:
Key
Shape
Dtype
Meaning
action
(T… See the full description on the dataset page: https://huggingface.co/datasets/Xia-2004/arx-left-cube.arxiv-clustering-s2s
ArXivHierarchicalClusteringS2S
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles from arxiv. Clustering of 30 sets, either on the main or secondary category
Task category
t2c
Domains
Academic, Written
Reference
https://www.kaggle.com/Cornell-University/arxiv
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["ArXivHierarchicalClusteringS2S"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/arxiv-clustering-s2s.ML-ArXiv-PapersThis dataset contains the subset of ArXiv papers with the "cs.LG" tag to indicate the paper is about Machine Learning.
The core dataset is filtered from the full ArXiv dataset hosted on Kaggle: https://www.kaggle.com/datasets/Cornell-University/arxiv. The original dataset contains roughly 2 million papers. This dataset contains roughly 100,000 papers following the category filtering.
The dataset is maintained by with requests to the ArXiv API.
The current iteration of the dataset only contains… See the full description on the dataset page: https://huggingface.co/datasets/CShorten/ML-ArXiv-Papers.arxiv_papers
ArXiv Papers
Description
ArXiv is an online open-access repository of over 2.4 million scholarly papers covering fields such as computer science, mathematics, physics, quantitative biology, economics, and more.
When uploading papers, authors can choose from a variety of licenses.
This dataset includes text from all papers uploaded under CC BY, CC BY-SA, and CC0 licenses through a three-step pipeline:
first, the latex source files for openly licensed papers were… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/arxiv_papers.arxiv-clustering-p2p
ArXivHierarchicalClusteringP2P
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles+abstract from arxiv. Clustering of 30 sets, either on the main or secondary category
Task category
t2c
Domains
Academic, Written
Reference
https://www.kaggle.com/Cornell-University/arxiv
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/arxiv-clustering-p2p.arxiv-abstracts-2021
Dataset Card for arxiv-abstracts-2021
Dataset Summary
A dataset of metadata including title and abstract for all arXiv articles up to the end of 2021 (~2 million papers).
Possible applications include trend analysis, paper recommender engines, category prediction, knowledge graph construction and semantic search interfaces.
In contrast to arxiv_dataset, this dataset doesn't include papers submitted to arXiv after 2021 and it doesn't require any external download.… See the full description on the dataset page: https://huggingface.co/datasets/gfissore/arxiv-abstracts-2021.arxivmath
Homepage and repository
Homepage: https://matharena.ai/
Repository: https://github.com/eth-sri/matharena
Dataset Summary
This dataset contains the questions from ArXivMath used for the MathArena Leaderboard
Data Fields
The dataset contains the following fields:
problem_idx (int64): Problem index within the corresponding MathArena benchmark.
answer (string): Gold final answer.
problem_type (list[string]): Problem type/category labels.
source (float64): arXiv… See the full description on the dataset page: https://huggingface.co/datasets/MathArena/arxivmath.arxiv_s2orc_parsed
Dataset Card for "ArtifactAI/arxiv_s2orc_parsed"
Dataset Description
https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_s2orc_parsed
Dataset Summary
AlgorithmicResearchGroup/arxiv_s2orc_parsed is a subset of the AllenAI S2ORC dataset, a general-purpose corpus for NLP and text mining research over scientific papers,
The dataset is filtered strictly for ArXiv papers, including the full text for each paper. Github links have been extracted… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_s2orc_parsed.