datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
ML-ArXiv-PapersThis dataset contains the subset of ArXiv papers with the "cs.LG" tag to indicate the paper is about Machine Learning.
The core dataset is filtered from the full ArXiv dataset hosted on Kaggle: https://www.kaggle.com/datasets/Cornell-University/arxiv. The original dataset contains roughly 2 million papers. This dataset contains roughly 100,000 papers following the category filtering.
The dataset is maintained by with requests to the ArXiv API.
The current iteration of the dataset only contains… See the full description on the dataset page: https://huggingface.co/datasets/CShorten/ML-ArXiv-Papers.arxiv_nlp
arXiv Abstracts
Abstracts for the cs.CL category of ArXiv between 1991 and 2024. This dataset was created as an instructional tool for the Clustering and Topic Modeling chapter in the upcoming
"Hands-On Large Language Models" book.
The original dataset was retrieved here.
This subset will be updated towards the release of the book to make sure it captures relatively recent articles in the domain.
arxivannotations
Title
Annotation
PDF
Latex
Axion bremsstrahlung from collisions of global strings
We calculate axion radiation emitted in the collision of two straight globalstrings. The strings are supposed to be in the unexcited ground state, to beinclined with respect to each other, and to move in parallel planes. Radiationarises when the point of minimal separation between the strings moves fasterthan light. This effect exhibits a typical Cerenkov nature. Surprisingly, itallows an alternative… See the full description on the dataset page: https://huggingface.co/datasets/Dan-Kos/arxivannotations.arXiv_latex
TeX data from arXiv
Using https://github.com/KyuDan1/TeX2Image code.
We have categories Math, Physics, Statistics, ComputerScience.
Domain
Size
Mathematics
4.22M
Computer Science
2.76M
Statistics
0.89M
Physics
0.78M
Total (unique)
7.17M
ann-arxiv-2m
2M Title-Abstract Arxiv Pairs
title_abstract.tsv data from Cornell University Arxiv Dataset, preprocessed and coverted to TSV.
title.e5-base-v2.fbin is a binary file with e5-base-v2 title embeddings.
abstract.e5-base-v2.fbin is a binary file with e5-base-v2 abstract embeddings.
pubmed_arxiv_abstracts_dataannotations_creators:
machine-generated
language:
en
license:
apache-2.0
multilinguality:
monolingual
task_categories:
classification
generation
pretty_name: PubMed_ArXiv_Abstracts
features:
name: abstr
dtype: string
name: title
dtype: string
name: journal
dtype: string
name: field
dtype: string
name: label_journal
dtype: int64
name: label_field
dtype: int64
hf-arxiv-url-bench
arXiv URL Extraction Benchmark & Longitudinal Corpus
Dataset Description
This repository hosts datasets designed to assess format-specific coverage gaps in URL extraction across the arXiv corpus. The collection facilitates large-scale reproducibility studies and evaluates how different document formats (LaTeX, HTML, XML, Markdown, TXT, PNG) impact automated extraction pipelines.
The repository is divided into two primary corpora:
The 200-Paper Benchmark: A… See the full description on the dataset page: https://huggingface.co/datasets/rochanaro/hf-arxiv-url-bench.ArXiv-10
ArXiv-10
"ArXiv-10" dataset consists of titles and abstracts extracted from 100k scientific papers on ArXiv, covering ten distinct research categories.
These categories includes subfields of computer science, physics, and mathematics.
To ensure consistency and manageability, the dataset consists of precisely 10k samples per category.
This dataset provides a practical resource for researchers and practitioners interested in LLM.
What is different about this dataset is the high… See the full description on the dataset page: https://huggingface.co/datasets/effectiveML/ArXiv-10.arxiv-software-engineering-datasetarXivTection
📄 arXivTection Dataset
The arXivTection dataset serves as a benchmark designed for the task of detecting pretraining data from Large Language models.
The dataset consists of 50 research papers extracted from arXiv.
25 published in 2023: Non-Training data, "label" column = 0.
25 published before 2022: Training data, "label" column = 1.
From each paper ≈ 30 passages are extracted. Each passage is paraphrased 3 times using the Language Model Claude v2.0.
The "Answer" column… See the full description on the dataset page: https://huggingface.co/datasets/avduarte333/arXivTection.word2vec_arxivearxiv_data_extendedarxiv-biology
Dataset Curators
The original data is maintained by ArXiv
Licensing Information
The data is under the Creative Commons CC0 1.0 Universal Public Domain Dedication
Citation Information
@misc{clement2019arxiv,
title={On the Use of ArXiv as a Dataset},
author={Colin B. Clement and Matthew Bierbaum and Kevin P. O'Keeffe and Alexander A. Alemi},
year={2019},
eprint={1905.00075},
archivePrefix={arXiv},
primaryClass={cs.IR}
}
arxiv-pii
ArXiv PII
Personally Identifiable Information (PII) from ArXiv front pages.
The dataset contains:
first pages of ArXiv PDFs converted to text
author related labels: NAME, EMAIL, PHONE
Available splits and data statistics:
split
pages
NAME
EMAIL
PHONE
2312
1749
9871
3993
39
2401
1312
7477
3039
36
Usage
ds_pages = load_dataset("sorokin/arxiv-pii", "pages", split="2312")
ds_labels = load_dataset("sorokin/arxiv-pii", "labels", split="2312")… See the full description on the dataset page: https://huggingface.co/datasets/sorokin/arxiv-pii.mathoverflow_text_arxiv_labelsDownloaded from https://archive.org/download/stackexchange
Used TexSoup to replace all text in math environments with [UNK]. For instance the text:
"The integral $\int_a^b f(x) \textrm{ d}x$ is easy to evaluate if..."
was replaced with
"The integral [UNK] is easy to evaluate if..."
Note: There is still some "ascii math". For instance, people sometimes write things like f: X --> Y. This is retained.
Concatenated title and body.
Some of these are "answer" posts rather than "question" posts.… See the full description on the dataset page: https://huggingface.co/datasets/stevengubkin/mathoverflow_text_arxiv_labels.ARxiv_Metadata_50k
arXiv Computer Science Dataset Description
Overview
This dataset is a filtered subset of arXiv papers focusing exclusively on Computer Science domains. It contains 46,328 academic papers with their metadata and classifications.
Dataset Structure
The dataset contains 7 columns:
id: Unique identifier for each arXiv paper
submitter: The individual who submitted the paper to arXiv
title: The title of the research paper
categories: Original arXiv category codes… See the full description on the dataset page: https://huggingface.co/datasets/AyoubChLin/ARxiv_Metadata_50k.Last-Week-on-ML-ArXivPlease check here to see when the dataset was last updated.
Last Updated July 12th, 2022 arxiv_small_nougat
Dataset Description
The "arxiv_small_nougat" dataset is a collection of 108 recent papers sourced from arXiv, focusing on topics related to Large Language Models (LLM) and Transformers. These papers have been meticulously processed and parsed using Meta's Nougat model, which is specifically designed to retain the integrity of complex elements such as tables and mathematical equations.
Data Format
The dataset contains the parsed content of the selected papers, with special… See the full description on the dataset page: https://huggingface.co/datasets/deep-learning-analytics/arxiv_small_nougat.arxiv_qa
Arxiv Paper Generative Question Answering
Dataset Summary
This dataset is made using ChatGPT (text-davinci-003) to generate Question/Answer pairs from Arxiv papers from this dataset
Data Fields
TextID: references the datarow (paper) in the arxiv summarizer dataset
Question: question based on the text
Response: answer
Text: Full text with the paper as 'context:' and and the question appended as 'question:'. Used for generative question answering usign language… See the full description on the dataset page: https://huggingface.co/datasets/TitanMLData/arxiv_qa.GraphTranslator-arxivarxiv-metadata-10000dynamic_topic_modeling_arxiv_abstractsarxiv_embedding_splitArXiv-ML-Title-EmbeddingsThis dataset contains embeddings of the titles of ArXiv Machine Learning papers.
The embeddings are produced from sentence-transformers/paraphrase-MiniLM-L6-v2. The model can be accessed here: HuggingFace Sentence Transformers
The original dataset before embedding can be accessed here: ML ArXiv Papers
Arxiv_Train4000_Val2000_Test2000AI_Articles_Scraped_from_arXiv-Semantic_Scholar
📘 AI Articles Scraped from arXiv & Semantic Scholar
🧩 Description
This dataset contains information on articles related to major AI conferences such as AAAI, NeurIPS, IJCAI, ICML, ICLR, collected through scraping from ArXiv and Semantic Scholar.It is intended to be used as a training dataset for various model training tasks and other desired uses.
📂 File Structure
File
Description
AI_Titles_v2025.csv
Main dataset
README.md
This file… See the full description on the dataset page: https://huggingface.co/datasets/d-e-c-d/AI_Articles_Scraped_from_arXiv-Semantic_Scholar.ArXiv-ML-Abstract-EmbeddingsThis dataset contains embeddings of the abstracts of ArXiv Machine Learning papers.
The embeddings are produced from sentence-transformers/paraphrase-MiniLM-L6-v2. The model can be accessed here: HuggingFace Sentence Transformers
The original dataset before embedding can be accessed here: ML ArXiv Papers
arxiv_cs_categoryThis is the same datasets as in Kaggle (https://www.kaggle.com/datasets/Cornell-University/arxiv)
Only selected for CS category
Arxiv_2000Arxiv_train_test