datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
10k_prompts_ranked
Dataset Card for 10k_prompts_ranked
10k_prompts_ranked is a dataset of prompts with quality rankings created by 314 members of the open-source ML community using Argilla, an open-source tool to label data. The prompts in this dataset include both synthetic and human-generated prompts sourced from a variety of heavily used datasets that include prompts.
The dataset contains 10,331 examples and can be used for training and evaluating language models on prompt ranking tasks. The… See the full description on the dataset page: https://huggingface.co/datasets/data-is-better-together/10k_prompts_ranked.newyorker_caption_ranking
New Yorker Caption Ranking Dataset
Dataset Descriptions
Homepage: https://nextml.github.io/caption-contest-data/
Repository: https://github.com/yguooo/cartoon-caption-generation
Paper: Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning
Point of Contact: yguo@cs.wisc.edu
Dataset Summary
We present a novel multimodal preference dataset for creative tasks, consisting of over 250 million human ratings on more than 2.2… See the full description on the dataset page: https://huggingface.co/datasets/yguooo/newyorker_caption_ranking.wikipedia-ja-20230101
range3/wikipedia-ja-20230101
This dataset consists of a parquet file from the wikipedia dataset with only Japanese data extracted. It is generated by the following python code.
このデータセットは、wikipediaデータセットの日本語データのみを抽出したparquetファイルで構成されます。以下のpythonコードによって生成しています。
import datasets
dss = datasets.load_dataset(
"wikipedia",
language="ja",
date="20230101",
beam_runner="DirectRunner",
)
for split,ds in dss.items():
ds.to_parquet(f"wikipedia-ja-20230101/{split}.parquet")
cc100-ja
range3/cc100-ja
This dataset consists of parquet files from the cc100 dataset with only the Japanese language extracted and sharded.
このデータセットは、cc100データセットの日本語のみを抽出し、シャーディングしたparquetファイルで構成されます。
wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/rancper/wikipedia.random-small-github-repositories
random-small-github-repositories
A collection of 5,613 small-to-medium open-source GitHub repositories, packaged as zipped archives alongside a metadata CSV. Intended as a seed dataset for code retrieval, context engineering, and SWE-bench-style dataset construction tasks.
Contents
seed_small_repos.csv — metadata for each repo (owner, repo_name, stars, license, repo_hash)
repos-zipped/ — one .zip per repo, named {repo_hash}.zip
unzipper.py - unzipping python… See the full description on the dataset page: https://huggingface.co/datasets/AmanPriyanshu/random-small-github-repositories.random-python-github-repositories
random-python-github-repositories
A collection of 1650 open-source Python GitHub repositories, packaged as zipped archives alongside a metadata CSV. Intended as a seed dataset for code retrieval, context engineering, and SWE-bench-style dataset construction tasks. All repos contain 250+ .py files.
Contents
repos_meta_data.csv — metadata for each repo (owner, repo_name, stars, license, py_file_count, alpha_hash)
repos-zipped/ — one .zip per repo, named… See the full description on the dataset page: https://huggingface.co/datasets/AmanPriyanshu/random-python-github-repositories.RefineCut-Bench
RefineCut-Bench
A planning-level benchmark for executable video-editing planning (EMNLP 2026, Plans You Can Check:
Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing). A task gives a planner a
brief, a real clip pool with schema-constrained captions and metadata, optional music metadata with beat
tracks, the current timeline state, and an explicit constraint ledger; the planner emits a RefinePatch
(RFC 6902-style JSON Patch over a typed timeline)… See the full description on the dataset page: https://huggingface.co/datasets/Randallhy/RefineCut-Bench.cancer-knowledge-base
Cancer Knowledge Base — the open, verified oncology KB for RAG & LLM evaluation
The only open CC-BY-4.0 oncology knowledge base that combines:
110/110 trials cited with PMID + NCT + PubMed/ClinicalTrials.gov URLs, and 32 prognosis
rows linked to verified SEER 2016–2022 references — no LLM-synthetic dataset has this.
A provable 152-question MCQ benchmark — every answer derives from this KB's own structured
data and carries a citation + golden docs, so it is open-book verifiable… See the full description on the dataset page: https://huggingface.co/datasets/ranjithraj/cancer-knowledge-base.leandojo-benchmark-4-randomThe random split of LeanDojo Benchmark 4.
Source data: https://zenodo.org/record/12740403/files/leandojo_benchmark_4.tar.gz
task722_mmmlu_answer_generation_random_topic
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task722_mmmlu_answer_generation_random_topic
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task722_mmmlu_answer_generation_random_topic.wiki40b-ja
range3/wiki40b-ja
This dataset consists of three parquet files from the wiki40b dataset with only Japanese data extracted. It is generated by the following python code.
このデータセットは、wiki40bデータセットの日本語データのみを抽出した3つのparquetファイルで構成されます。以下のpythonコードによって生成しています。
import datasets
dss = datasets.load_dataset(
"wiki40b",
"ja",
beam_runner="DirectRunner",
)
for split,ds in dss.items():
ds.to_parquet(f"wikipedia-ja-20230101/{split}.parquet")
Java_method2test_chatml
Java Method to Test ChatML
This dataset is based on the methods2test dataset from Microsoft. It follows the ChatML template format: [{'role': '', 'content': ''}, {...}].
Originally, methods2test contains only Java methods at different levels of granularity along with their corresponding test cases. The different focal method segmentations are illustrated here:
To simulate a conversation between a Java developer and an AI assistant, I introduce two key parameters:
The prompt… See the full description on the dataset page: https://huggingface.co/datasets/random-long-int/Java_method2test_chatml.SiPaKosa-Sent
SiPaKosa: Sinhala-Pali Buddhist Corpus
A comprehensive corpus of canonical and classical Buddhist texts in Sinhala and Pali, compiled from historical archives and web-scraped canonical scriptures.
This is the sentence-level version of the SiPaKosa dataset.
Where SiPaKosa contains book level text, this dataset has all sentences by book.
Related dataset (book-level): RaniduG/SiPaKosa
Dataset Statistics
Total Sentences: 786,344
Sinhala Sentences: 465,539 (59.2%)
Mixed… See the full description on the dataset page: https://huggingface.co/datasets/RaniduG/SiPaKosa-Sent.random-captions-10mRandomly generated captions using tokenization templates and lists.
.,|,. is the caption delimiter, so split accordingly.
coding-sessions
RangaPrasath Coding Sessions
Dataset Description
This dataset contains 73 real coding sessions from OpenAI Codex, exported using the pi-brain tool as part of the Sybil Solutions 20T Session Data Drive.
Source Tool
Exporter: pi-brain v0.1.0
Source: OpenAI Codex (~/.codex/sessions/)
Export date: 2026-05-16
What's Included
73 sessions with full trajectories
22,528 messages across all sessions
Complete tool calls, code edits, test output, and review… See the full description on the dataset page: https://huggingface.co/datasets/RangaPrasath/coding-sessions.nepal-section-wise-act-datasets
Nepal Section-wise Act Datasets
Dataset Description
This dataset contains section-wise legal acts and laws of Nepal, organized for easy access and analysis. It is designed to support legal research, natural language processing (NLP) tasks, and the development of legal tech applications in Nepal.
Note: This dataset is released for research purposes only. Any other unwanted use can lead to the violation of the intended terms of use and may result in legal action or… See the full description on the dataset page: https://huggingface.co/datasets/ranjitraut/nepal-section-wise-act-datasets.human_rank_eval
Dataset Card for HumanRankEval
This dataset supports the NAACL 2024 paper HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants.
Dataset Description
Language models (LMs) as conversational assistants recently became popular tools that help people accomplish a variety of tasks. These typically result from adapting LMs pretrained on general domain text sequences through further instruction-tuning and possibly preference optimisation methods. The evaluation… See the full description on the dataset page: https://huggingface.co/datasets/huawei-noah/human_rank_eval.lean4-stat-learning-theory-random
A Large-Scale Lean 4 Dataset on Statistical Learning Theory
We present a high-quality, human-verified, large-scale Lean 4 dataset, extracted from our formalization of Statistical Learning Theory (SLT). We present the first comprehensive Lean 4 formalization of SLT grounded in empirical process theory. Our end-to-end formal infrastructure implement the missing contents in latest Lean 4 Mathlib library, including a complete development of Gaussian Lipschitz concentration… See the full description on the dataset page: https://huggingface.co/datasets/yuanhezhang/lean4-stat-learning-theory-random.task1315_find_range_array
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1315_find_range_array
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks}… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1315_find_range_array.PersonaGen-Enterprise
PersonaGen-Enterprise: B2B Buying Intelligence Dataset
5,000 enterprise buyer personas with full buying committee modeling across 15 industries, 3 company sizes, and 42 buying roles. Plus 47K real search queries, 7.5K competitive brand queries, and multi-model agreement scores.
Built by Rankfor.AI, the AI Visibility Intelligence platform. This dataset powers research into how enterprise buyers search for, evaluate, and select B2B technology vendors.
Enterprise… See the full description on the dataset page: https://huggingface.co/datasets/rankfor/PersonaGen-Enterprise.rank1-R1-MSMARCO
rank1-R1-MSMARCO: Reasoning Outputs from MS MARCO Dataset
📄 Paper | 🚀 GitHub Repository
This dataset contains outputs from Deepseek's R1 model on the MS MARCO passage dataset, used to train rank1. It showcases the reasoning chains and relevance judgments generated when determining document relevance for information retrieval queries.
Dataset Description
The rank1-R1-MSMARCO dataset consists of reasoning chains and relevance judgments produced on the MS MARCO passage… See the full description on the dataset page: https://huggingface.co/datasets/jhu-clsp/rank1-R1-MSMARCO.low-quality-random-sft-data-I-had-laying-around
low-quality-random-sft-data-I-had-laying-around
Exactly what it says on the tin. Random synthetic ChatML SFT scraps I had laying around on disk. Not curated. Not high quality. Possibly cursed. Useful if you want cheap filler / toy SFT data.
Configs
Config
Rows
What it is
counting
15,000
Letter counts, palindromes, tiny string puzzles
word-problems
19,587
Synthetic arithmetic word problems
math
213,693
Synthetic math Q&A ChatML (merged from several… See the full description on the dataset page: https://huggingface.co/datasets/hudsongouge/low-quality-random-sft-data-I-had-laying-around.finee-dataset
FinEE Dataset
Dataset Description
A comprehensive dataset for training financial entity extraction models on Indian banking messages. Contains 152,000+ samples covering SMS, emails, and transaction notifications from major Indian banks.
Languages
English (en) - 86%
Hindi (hi) - 3%
Tamil (ta) - 3%
Telugu (te) - 3%
Bengali (bn) - 3%
Kannada (kn) - 2%
Supported Transaction Types
UPI payments (PhonePe, GPay, Paytm)
NEFT/IMPS/RTGS transfers… See the full description on the dataset page: https://huggingface.co/datasets/Ranjit0034/finee-dataset.Mr-GSM8KView the project page:
https://github.com/dvlab-research/DiagGSM8K
see our paper at https://arxiv.org/abs/2312.17080
Description
In this work, we introduce a novel evaluation paradigm for Large Language Models,
one that challenges them to engage in meta-reasoning. Our paradigm shifts the focus from result-oriented assessments,
which often overlook the reasoning process, to a more holistic evaluation that effectively differentiates
the cognitive capabilities among models. For… See the full description on the dataset page: https://huggingface.co/datasets/Randolphzeng/Mr-GSM8K.PersonaGen-15K
PersonaGen-15K: Research Sample from PersonaGen-1M
14,955 anonymized buyer personas, a stratified subset of the 1,031,732-persona PersonaGen-1M corpus, drawn to preserve its market, intent and gender proportions.
This is the publicly available research sample of PersonaGen-1M, the largest dataset of structured AI-generated buyer personas designed for consumer information-seeking behavior research. The full corpus contains 1,031,732 personas carrying 19,416,821 behavioral… See the full description on the dataset page: https://huggingface.co/datasets/rankfor/PersonaGen-15K.Amazon-combined
Amazon Combined Dataset
E-commerce dataset that combines metadata, reviews, and sample question/answer pairs. combined.json contains the dataset and user2asin.json contains a file that maps user_id from reviews to an ASIN for capturing user preferences.
Data Fields
Field
Type
Explanation
main_category
str
Main category (i.e., domain) of the product.
title
str
Name of the product.
average_rating
float
Rating of the product shown on the product page.… See the full description on the dataset page: https://huggingface.co/datasets/randomath/Amazon-combined.rank1-training-data
rank1-training-data: Training Dataset for rank1 Reasoning Rerankers
📄 Paper | 🚀 GitHub Repository
This dataset contains the training data used to develop the rank1 family of reasoning rerankers with LLaMA Factory. It includes query-document pairs with relevance judgments and reasoning chains that guided the models to make binary relevance decisions.
Dataset Description
The rank1-training-data dataset is a comprehensive collection of training examples used to teach… See the full description on the dataset page: https://huggingface.co/datasets/jhu-clsp/rank1-training-data.agent-trajectories-swesmith-random-subset
Agent Trajectories: SWE-Smith Random Subset — Single Model
Summary
Full multi-turn agent trajectories collected from a random subset of SWE-Smith issues,
generated with a single model per trajectory (no branching or lookahead).
Contains a resolved column from SWE-bench evaluation.
Intended for SFT of agent models on diverse, synthetically generated coding tasks.
Data Collection
Each trajectory was produced by a mini-swe-agent run with an LLM router selecting… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/agent-trajectories-swesmith-random-subset.Assamese-Text-Dataset-45T-Tokens
I have massive Assamese Dataset nearly about 45.3T (45333004592600) Tokens
It has a lots of Assamese sentances from various sources, 99.9999% of the dataset are cleanned
just download the backup_data.tar.zst file and start using it.
happy training....
My email: ranjitdax89@gmail.com
At least share your opinion… or maybe a simple “thanks” 😄
Topic / Dataset
Tokens
Approx. Scale
Source
Poems Dataset
92.6K
0.0000926B… See the full description on the dataset page: https://huggingface.co/datasets/Ranjit89/Assamese-Text-Dataset-45T-Tokens.