datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
newswire
Dataset Card for NewsWire
Dataset Summary
NewsWire contains 2.7 million unique public domain U.S. news wire articles, written between 1878 and 1977. Locations in these articles are georeferenced, topics are tagged using customized neural topic classification, named entities are recognized, and individuals are disambiguated to Wikipedia using a novel entity disambiguation model.
Languages
English (en)
Dataset Structure
Each year in the dataset is… See the full description on the dataset page: https://huggingface.co/datasets/dell-research-harvard/newswire.arxiv_cplusplus_research_code
Dataset card for ArtifactAI/arxiv_cplusplus_research_code
Dataset Description
https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_cplusplus_research_code
Dataset Summary
ArtifactAI/arxiv_python_research_code contains over 10.6GB of source code files referenced strictly in ArXiv papers. The dataset serves as a curated dataset for Code LLMs.
How to use it
from datasets import load_dataset
# full dataset (10.6GB of data)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_cplusplus_research_code.bankertoolbench
BankerToolBench
BankerToolBench is a benchmark of 100 end-to-end investment banking tasks for
evaluating AI agents. Each task mirrors real junior-banker work — building
financial models, preparing pitch decks, writing memos — and produces multi-file
deliverables (Excel, PowerPoint, Word) that are scored against expert-authored
rubrics.
The benchmark was developed with 502 investment bankers from firms including
Goldman Sachs, JPMorgan, Evercore, and others. Human completion time… See the full description on the dataset page: https://huggingface.co/datasets/handshake-ai-research/bankertoolbench.commit-chronicle
📜 CommitChronicle 🔮
This is the dataset for commit message generation (and/or completion), introduced in the paper "From Commit Message Generation to History-Aware Commit Message Completion", ASE 2023.
Its key features:
large-scale and multilingual: contains 10.7M commits from 11.9k GitHub repositories in 20 programming languages;
diverse: avoids restrictive filtering on commit messages or commit diffs structure;
suitable for experiments with commit history: provides metadata… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/commit-chronicle.VAKRA
🔷 VAKRA: A Benchmark for Evaluating Multi-Hop, Multi-Source Tool-Calling Capabilities in AI Agents
VAKRA (eValuating API and Knowledge Retrieval Agents using multi-hop, multi-source dialogues) is a tool-grounded, executable benchmark designed to evaluate how well AI agents reason end-to-end in enterprise-like settings.
Rather than testing isolated skills, VARKA measures compositional reasoning across APIs and documents, using full execution traces to assess whether agents can… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/VAKRA.researchscope-papers
ResearchScope Papers
Open CS research paper dataset maintained by ResearchScope.
Updated automatically via GitHub Actions.
Quick start
from datasets import load_dataset
ds = load_dataset("kishormorol/researchscope-papers", "papers", split="train")
print(ds[0])
See Usage below for per-source splits, instruction-tuning, and the per-section fine-tuning data.
Stats
34,903 papers (raw metadata) — 9,903 arXiv · 20,000 conference · 5,000 journal
174,067… See the full description on the dataset page: https://huggingface.co/datasets/kishormorol/researchscope-papers.lca-bug-localization
🏟️ Long Code Arena (Bug localization)
This is the benchmark for the Bug localization task as part of the
🏟️ Long Code Arena benchmark.
The bug localization problem can be formulated as follows: given an issue with a bug description and a repository snapshot in a state where the bug is reproducible, identify the files within the repository that need to be modified to address the reported bug.
The dataset provides all the required components for evaluation of bug localization… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/lca-bug-localization.Scientific_Research_Tokenized
NexaSci Scientific Research Tokenized
This dataset repository now holds the active NexaSci scientific pretraining reservoir, the NexaMat controller fine-tuning pack, and archived legacy reservoir builds. The current production reservoir is the 10B-token Apache Arrow release under nexasci_reservoir_v3_10b_prod_rust/.
Current Status
The active large-scale training artifact is:
nexasci_reservoir_v3_10b_prod_rust/
It was produced from the NexaSci 10B data-engineering campaign… See the full description on the dataset page: https://huggingface.co/datasets/AethronPhantom/Scientific_Research_Tokenized.Nepali-Text-Corpus
Nepali Text Corpus
Overview
Nepali-Text-Corpus is a comprehensive collection of approximately 6.4 million articles in the
Nepali language. This dataset is the largest text dataset on Nepali Language. It encompasses a
diverse range of text types, including news articles, blogs, and more, making it an invaluable
resource for researchers, developers, and enthusiasts in the fields of Natural Language Processing (NLP)
and computational linguistics.
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/IRIIS-RESEARCH/Nepali-Text-Corpus.qwen3.8-max-glm5.2-kimi-k3-distillation
Multi-Teacher Distillation Dataset (57,937 traces)
A quality-filtered, deduplicated, multi-teacher SFT corpus combining traces from three frontier models across math, code, reasoning, instruction-following, tool-use, science, long-context, multilingual, and creative dialogue domains.
Teachers
Teacher
Provider
Traces
Qwen3.8-Max-Preview
Alibaba Cloud Model Studio
48,283
GLM-5.2
Z.AI Coding Plan
5,307
Kimi Code K3
Moonshot AI (Kimi)
4,347… See the full description on the dataset page: https://huggingface.co/datasets/p-research/qwen3.8-max-glm5.2-kimi-k3-distillation.python-text-copilot-training-instruct-ai-research-2024-02-03
Python Copilot Instructions on How to Code using Alpaca and Yaml
Training and test datasets for building coding multimodal models that understand how to use the open source GitHub projects for the Agora Open Source AI Research Lab:
Agora GitHub Organization
Agora Hugging Face
This dataset is the 2024-02-03 update for the matlok python copilot datasets. Please refer to the Multimodal Python Copilot Training Overview for more details on how to use this dataset.
Details… See the full description on the dataset page: https://huggingface.co/datasets/matlok/python-text-copilot-training-instruct-ai-research-2024-02-03.arxiv_research_code
Dataset Card for "AlgorithmicResearchGroup/arxiv_research_code"
Dataset Description
https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_research_code
Dataset Summary
ArtifactAI/arxiv_research_code contains over 21.8GB of source code files referenced strictly in ArXiv papers. The dataset serves as a curated dataset for Code LLMs.
How to use it
from datasets import load_dataset
# full dataset (21.8GB of data)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_research_code.ResearchMath-14k
ResearchMath-14k
ResearchMath-14k is a collection of 14,056 research-level mathematical problem records extracted from papers, open-problem lists, workshop sheets, and related academic sources. Each record contains the original extracted question, a rewritten self-contained problem statement, taxonomy labels, and open-status metadata.
Paper: ResearchMath-14K: Scaling Research-Level Mathematics via Agents
Load
from datasets import load_dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/amphora/ResearchMath-14k.arxiv_python_research_code
Dataset Card for "ArtifactAI/arxiv_python_research_code"
Dataset Description
https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_python_research_code
Dataset Summary
AlgorithmicResearchGroup/arxiv_python_research_code contains over 4.13GB of source code files referenced strictly in ArXiv papers. The dataset serves as a curated dataset for Code LLMs.
How to use it
from datasets import load_dataset
# full dataset (4.13GB of data)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_python_research_code.Funcdex-MT-Function-Calling
Funcdex-MT-Function-Calling Dataset
Funcdex-MT-Function-Calling is a multi-turn function calling dataset designed for training language models to interact with real-world tools and APIs. The dataset contains 1,787 conversations covering 10 individual toolkits and 5 multi-toolkit bundles, with comprehensive system prompts and realistic multi-turn interactions.The code used to generate the dataset can be found here.
Models trained on this dataset have excellent… See the full description on the dataset page: https://huggingface.co/datasets/prem-research/Funcdex-MT-Function-Calling.litcoin-proof-of-research
LITCOIN Proof-of-Research Corpus
191,484,662 AI research submissions, produced by 81,224 anonymous contributors and 470 model
variants competing against each other, every row executed in a sandbox and scored.
This is the complete output of the LITCOIN protocol, which ran on Base from March to August 2026.
Autonomous AI agents were paid in a permissionless token to solve real optimization problems across
32 domains. The protocol was discontinued on 20 August 2026. This dataset is… See the full description on the dataset page: https://huggingface.co/datasets/tekkaadan/litcoin-proof-of-research.nestful
NESTFUL: Nested Function-Calling Dataset
NESTFUL is a benchmark to evaluate LLMs on nested sequences of API calls, i.e., sequences where the output of one API call is passed as input to
a subsequent call.
The NESTFUL dataset includes over 1800 nested sequences from two main areas: mathematical reasoning and coding tools. The mathematical reasoning portion is generated from
the MathQA dataset, while the coding portion is generated from the
StarCoder2-Instruct dataset.
All… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/nestful.ATLAS-Finance
ATLAS Finance
A benchmark of 100 expert-level tasks inside 13 realistic financial firm environments, packaged in the Harbor RLE format.
Each task drops an AI agent into a Linux workstation with a
persistent multi-app world — inbox, chat, calendar, virtual data room, drive,
wiki — and asks the agent to produce the same deliverable a financial professional would be responsible for:
an Excel workbook containing the model and supporting analysis.
Here we provide the data for this… See the full description on the dataset page: https://huggingface.co/datasets/handshake-ai-research/ATLAS-Finance.noteflow-research-pilots
Keep the failed attempts. Check the artifact.
Versioned public development evidence from Robot Reel × Skills Anywhere × EvalArc, recorded 14 September 2026 on an NVIDIA L40S, with separate scripted Harbor controls on CPU and separate GPU context-control and agent-requested MCP handoff cohorts recorded 19 September 2026. This is an inspectable engineering casebook, not a held-out benchmark or training corpus with established efficacy.
Configuration
Actual experiment
What… See the full description on the dataset page: https://huggingface.co/datasets/glayguo/noteflow-research-pilots.fable5-traces-sft
Fable 5 Traces — Unified SFT / Self-Distillation Dataset
A cleaned, unified, PII-scrubbed corpus of Claude Fable 5 agent traces in
OpenAI-style chat format, plus a working on-policy self-distillation (SDFT)
training scaffold.
Composition
Source
Conversations
Claude Code raw agentic sessions
18
CoT distillation records
4,665
Unique conversations (post-dedup)
4,683
Split deterministically by content hash: train 4,442 / validation 241.
The raw… See the full description on the dataset page: https://huggingface.co/datasets/Swarm-AI-Research/fable5-traces-sft.AttaQ
AttaQ Dataset Card
The AttaQ red teaming dataset, consisting of 1402 carefully crafted adversarial questions, is designed to evaluate Large Language Models (LLMs) by assessing their tendency to generate harmful or undesirable responses.
It may serve as a benchmark to assess the potential harm of responses produced by LLMs.
The dataset is categorized into seven distinct classes of questions: deception, discrimination, harmful information, substance abuse, sexual content, personally… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/AttaQ.ResearchMath-Reasoning-194K
ResearchMath-Reasoning-194K
ResearchMath-Reasoning-194K is a collection of 193,938 long-form reasoning traces and solutions for research-level mathematical problems, released alongside ResearchMath-14k as part of the same paper. While ResearchMath-14k provides the curated problem statements, this dataset provides model-generated solution attempts: each record contains a self-contained problem statement, a long chain-of-thought reasoning trace, and a final response.
Paper:… See the full description on the dataset page: https://huggingface.co/datasets/amphora/ResearchMath-Reasoning-194K.arxiv_deep_learning_python_research_code
ArXiv Deep Learning Python Research Code
A curated corpus of Python source code files extracted from GitHub repositories referenced in ArXiv papers. Contains 391,496 files (1.49 GB) filtered to deep learning frameworks, designed for training and evaluating Code LLMs on research-grade code.
Dataset Summary
Statistic
Value
Total files
391,496
Total size
1.49 GB
Source repos
34,099
Time span
ArXiv inception through July 2023
Dataset Structure… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_deep_learning_python_research_code.openresearcher-sft-deep-research-cleaned
OpenResearcher SFT DeepResearch — Parquet Mirror
This is a re-hosted copy of the tool-reasoning SFT deep-research dataset by Aman Priyanshu, itself a cleaned/restructured version of the OpenResearcher Dataset from TIGER-AI-Lab.
Why this repo exists: the source wasn't laid out as ready-to-download Parquet files. This mirror simply stores the data as plain seed_*.parquet files so you can grab the whole dataset or a single segment easily. No changes were made to the content — all… See the full description on the dataset page: https://huggingface.co/datasets/DanielTobi0/openresearcher-sft-deep-research-cleaned.OR-Clarify
OR-Clarify
📄 Paper: Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization
OR-Clarify is a benchmark for testing whether an agent asks the right questions before formulating an optimization model.
Most optimization benchmarks give an agent a complete problem statement. OR-Clarify instead starts with an incomplete business brief. The agent must identify missing requirements that could change the optimization formulation, ask for the relevant… See the full description on the dataset page: https://huggingface.co/datasets/AIOR-Research/OR-Clarify.python-text-copilot-training-instruct-ai-research
Building an AI Copilot Dataset to help keep up with Leading AI Research
This is a specialized, instruction dataset for training python coding assistants on how to code from leading AI/ML open source repositories (2.3M coding samples).
This dataset is a subset of the matlok python copilot datasets. Please refer to the Multimodal Python Copilot Training Overview for more details on how to use this dataset.
Details
This dataset holds the latest coding changes from >1159… See the full description on the dataset page: https://huggingface.co/datasets/matlok/python-text-copilot-training-instruct-ai-research.python-text-copilot-training-instruct-ai-research-2024-02-10
Python Copilot Instructions on How to Code using Alpaca and Yaml
Training and test datasets for building coding multimodal models that understand how to use the open source GitHub projects for the multimodal Qwen AI project:
Qwen
Qwen Agent
Qwen VL Chat
Qwen Audio
This dataset is the 2024-02-10 update for the matlok python copilot datasets. Please refer to the Multimodal Python Copilot Training Overview for more details on how to use this dataset.
Details
Each row… See the full description on the dataset page: https://huggingface.co/datasets/matlok/python-text-copilot-training-instruct-ai-research-2024-02-10.python-text-copilot-training-instruct-ai-research-2024-02-11
Python Copilot Instructions on How to Code using Alpaca and Yaml
Training and test datasets for building coding multimodal models that understand how to use the open source GitHub projects for the Autogen and multimodal Qwen AI project:
Qwen
Qwen Agent
Qwen VL Chat
Qwen Audio
This dataset is the 2024-02-11 update for the matlok python copilot datasets. Please refer to the Multimodal Python Copilot Training Overview for more details on how to use this dataset.
Details… See the full description on the dataset page: https://huggingface.co/datasets/matlok/python-text-copilot-training-instruct-ai-research-2024-02-11.agent-trajectories-swe-bench-test-minus-verified
Agent Trajectories: SWE-bench Test \ Verified — Mixed Teachers (gpt-5.2 / gpt-5-mini)
Summary
Full multi-turn agent trajectories collected from the SWE-bench Test minus Verified split
(i.e., SWE-bench Test instances that are not part of SWE-bench Verified).
Intended for SFT of agent models on coding tasks.
Data Collection
Each trajectory was produced by a GT-aware lookahead agent that, at every turn:
Sampled a candidate response from both gpt-5.2 and… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/agent-trajectories-swe-bench-test-minus-verified.Split-IFEval
Split IFEval
This dataset modifies the Instruction-Following Eval (IFEval) benchmark to split apart the task from the syntactic instructions in addition to fixing errors in the original dataset.
It enables the use of research methods like attention steering that require access to the instruction text.
To load the dataset, run:
from datasets import load_dataset
split_ifeval = load_dataset("ibm-research/Split-IFEval")
Dataset Structure
Each entry in the dataset… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/Split-IFEval.