datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
arxiv-complete
arXiv Complete Corpus
A snapshot of arXiv's metadata, version history, submission files and rendered
documents. It covers 3,148,796 papers and includes file contents, paths, sizes
and SHA-256 digests. Metadata comes from arXiv's OAI-PMH arXivRaw interface;
files come from the GCS mirror, S3 source archives and direct PDF fetches.
This release holds a PDF for 99.47% of papers and 99.54% of versions reported
with a non-zero submission size. It is a one-off snapshot; coverage gaps… See the full description on the dataset page: https://huggingface.co/datasets/secemp9/arxiv-complete.arxiv_cplusplus_research_code
Dataset card for ArtifactAI/arxiv_cplusplus_research_code
Dataset Description
https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_cplusplus_research_code
Dataset Summary
ArtifactAI/arxiv_python_research_code contains over 10.6GB of source code files referenced strictly in ArXiv papers. The dataset serves as a curated dataset for Code LLMs.
How to use it
from datasets import load_dataset
# full dataset (10.6GB of data)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_cplusplus_research_code.ArXivSignals-DeepSummaries
ArXivSignals DeepSummaries — Agent-Built Visual Paper Explainers
A continuously-updated, day-partitioned dataset of deep, visual summaries of
arXiv papers, each built by a coding agent working inside the paper's own
LaTeX source: the agent reads the full text, authors an editorial narrative as
a structured content spec, and the paper's real figures and tables
(extracted and rendered from the LaTeX, web-optimized) ride along as an
embedded, variable-length image array. The… See the full description on the dataset page: https://huggingface.co/datasets/taesiri/ArXivSignals-DeepSummaries.32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by Qwen3-32B
with thinking mode off. Each thought is a few dense sentences of reasoning about the next
8 tokens after a cut, written from the document prefix alone — the generator never sees the
continuation. Stored thought_text includes the <thought>/</thought> wrapper.
The 4B parity counterpart is… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.arxiv_research_code
Dataset Card for "AlgorithmicResearchGroup/arxiv_research_code"
Dataset Description
https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_research_code
Dataset Summary
ArtifactAI/arxiv_research_code contains over 21.8GB of source code files referenced strictly in ArXiv papers. The dataset serves as a curated dataset for Code LLMs.
How to use it
from datasets import load_dataset
# full dataset (21.8GB of data)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_research_code.arxiv_python_research_code
Dataset Card for "ArtifactAI/arxiv_python_research_code"
Dataset Description
https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_python_research_code
Dataset Summary
AlgorithmicResearchGroup/arxiv_python_research_code contains over 4.13GB of source code files referenced strictly in ArXiv papers. The dataset serves as a curated dataset for Code LLMs.
How to use it
from datasets import load_dataset
# full dataset (4.13GB of data)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_python_research_code.4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
Qwen3-4B-Instruct-2507. Each thought is a few dense sentences of reasoning about the next
8 tokens after a cut, written from the document prefix alone — the generator never sees the
continuation. Stored thought_text includes the <thought>/</thought> wrapper.
This is the small-generator parity counterpart of… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.arxiv_deep_learning_python_research_code
ArXiv Deep Learning Python Research Code
A curated corpus of Python source code files extracted from GitHub repositories referenced in ArXiv papers. Contains 391,496 files (1.49 GB) filtered to deep learning frameworks, designed for training and evaluating Code LLMs on research-grade code.
Dataset Summary
Statistic
Value
Total files
391,496
Total size
1.49 GB
Source repos
34,099
Time span
ArXiv inception through July 2023
Dataset Structure… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_deep_learning_python_research_code.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled
Every non-first chunk of every document carries a thought: the gpt-5.6-luna reasoning thought
where one was generated, and a content-free pause thought everywhere else.
luna chunk <|reserved_special_token_1|> luna reasoning <|reserved_special_token_2|>
filler chunk <|reserved_special_token_1|> 256x <|reserved_special_token_0|> <|reserved_special_token_2|>
The filler is 258 tokens. Chunk 0 is excluded… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled.4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/4B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as
Qwen/Qwen3-4B-Base… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.arxiv-qa-thinking
ArXiv Q&A with Thinking Dataset
This dataset contains question-answer pairs generated by MiniMax-M2.1 based on academic articles from PursuitOfDataScience/arxiv-llama4-maverick-abstract.
Dataset Description
For each academic article, the model generates:
Thinking process: The model's reasoning wrapped in <think> tags
Question: An insightful question testing understanding of key concepts
Answer: A detailed answer based on the article content
Statistics… See the full description on the dataset page: https://huggingface.co/datasets/PursuitOfDataScience/arxiv-qa-thinking.Labeled-arXiv
Labeled-arXiv
Enriched arXiv paper metadata and derived author bibliometrics sourced from OpenAlex. Contains 2.93M papers and 1.72M authors across all arXiv subject areas.
Subsets
papers — 2.93M rows
Column
Type
Description
id
string
arXiv paper ID (e.g. 0704.0028)
submitter
string (nullable)
Name of the submitting author
authors
string
Raw author string from arXiv metadata
title
string
Paper title
comments
string (nullable)
Submitter… See the full description on the dataset page: https://huggingface.co/datasets/ShayManor/Labeled-arXiv.4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
Qwen3-4B-Instruct-2507.prestar-RL.reason-only.lr7e-7-kl0.step-2176 — Qwen3-4B-Instruct-2507 after RL against a frozen suffix conditional. Each thought is a few dense sentences of reasoning about the next 8 tokens after a cut, written from the document prefix alone — the generator never sees the… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
gpt-5.6-luna. Each thought is visible reasoning about the next 8 Qwen3 tokens after a cut,
written without ever seeing that continuation. Intended to be spliced into the document
before the chunk so a small model (Qwen3-4B-Base) can read the reasoning and predict the
chunk. Thoughts are strings, not token ids;… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A length-matched null control for JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids, stored directly in the
kv-tags-explained training format (there is no separate base repo). Every thought is this
one sentence, repeated:
We are thinking hard about what comes in the next 8 tokens by reasoning correctly and carefully about what comes before it in the document.
It is fluent, on-topic and… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids
4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
qwen3-4B-instruct-luna-distill-step2362
This generator is Qwen3-4B-Instruct-2507 SFT'd on the train set of gpt-5.6-luna thoughts (wandb run 3rkrz2vo, step 2362).
⚠️ Note that stride-8 chunks were also seen during training this generator. There is a concern that the distillation checkpoint may have memorized… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.tags
4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.tags
Tokenized, tag-wrapped form of JackHsieh/4B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as <|note|><thought><|/note|> and stored both as text (thought_text) and as
Qwen/Qwen3-4B-Base token ids (input_ids).
Longest thought: 510 tokens — a training run's max_thought_length must be at least
this.
Delimiter ids: <|note|> = 151669, <|/note|> = 151670.
A run must declare these under… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.tags.4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A pre-tokenized, tag-wrapped variant of JackHsieh/4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.qwen3-distill-3rkrz2vo.k-8.statml-arxiv-qwen3.qwen3-ids.tags
qwen3-distill-3rkrz2vo.k-8.statml-arxiv-qwen3.qwen3-ids.tags
Tokenized, tag-wrapped form of JackHsieh/qwen3-4B-instruct-luna-distill-step2362.reason-only.k-8.statml-arxiv-qwen3.
Each thought is wrapped as <|note|> … thought … <|/note|> and stored both as text (thought_text) and as
Qwen/Qwen3-4B-Base token ids (input_ids).
Longest thought: 770 tokens — a training run's max_thought_length must be at least
this.
Delimiter ids: <|note|> = 151669, <|/note|> = 151670.
A run must… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/qwen3-distill-3rkrz2vo.k-8.statml-arxiv-qwen3.qwen3-ids.tags.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A pre-tokenized, tag-wrapped variant of JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags
A pre-tokenized, tag-wrapped variant of JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text (thought_text) and as Qwen/Qwen3-4B-Base token ids (input_ids).
Tag ids and the spliced key are inserted as ids, never re-tokenized.
Delimiter ids: <|note|> = 151669… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.tags
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.tags
A pre-tokenized, tag-wrapped variant of JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>{thought_text}<|/note|>
and stored both as text (thought_text) and as Qwen/Qwen3-4B-Base token ids (input_ids).
Tag ids and the spliced key are inserted as ids, never re-tokenized.
Delimiter ids: <|note|> = 151669, <|/note|> = 151670. These are the first… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.tags.4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv
4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv
Reasoning about the next 8 Qwen3 tokens of stat.ML arXiv LaTeX, generated by
stock Qwen/Qwen3-4B-Instruct-2507 -- no fine-tuning, prompted with the reason-only-nothink.jinja reasoning template.
Documents: JackHsieh/statML-arxiv-40M-20M, 4096 Qwen3 tokens each.
Why this checkpoint: Stock Qwen3-4B-Instruct-2507 prompted with reason-only-nothink.jinja, a byte-identical copy of the reason-only.jinja template used for… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.arxiv-to-code-agentic-tool-calling
arxiv-to-code-agentic-tool-calling
Multi-turn tool-calling dataset where an assistant implements ML papers in PyTorch through file-creation and command-execution tool calls.
Built from lucidrains' (Phil Wang) open-source paper implementations. There are ~217 repositories on Codeberg, each implementing a different ML paper. This dataset reverse-engineers those into synthetic coding conversations.
What's in it
199 conversations, each covering one repository. Every… See the full description on the dataset page: https://huggingface.co/datasets/SultanR/arxiv-to-code-agentic-tool-calling.4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.32B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
32B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/32B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained.luna-reason-only.k-8.statml-arxiv-llama32
luna-reason-only.k-8.statml-arxiv-llama32
Prefix-only "thoughts" for next-token prediction on stat.ML arXiv LaTeX. Each thought is
visible reasoning about the next 8 Llama-3.2 tokens after a cut, written without ever
seeing that continuation. Intended to be spliced into the document before the chunk so a
small model (Llama 3.2 3B base) can read the reasoning and predict the chunk.
Complete: every designated chunk has a thought.
split
thoughts
coverage
documents… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A pre-tokenized, tag-wrapped variant of JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as <|reserved_special_token_1|> … thought … <|reserved_special_token_2|> … last prefix token and stored both as text (thought_text) and as Llama 3.2
token ids (input_ids).
The trailing token is the document token immediately before the cut (input_ids[chunk_start_index - 1]), copied from the document rather… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo.