datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
arctic
Arctic Shift Reddit Archive
Every Reddit comment and submission since 2005, organized as monthly Parquet shards
What is it?
The full Reddit archive from Arctic Shift, converted to Parquet and hosted here for easy access. Covers every public subreddit from 2005-12 through 2026-02.
Right now the archive has 15.7B items (12.9B comments, 2.8B submissions) in 1.3 TB of compressed Parquet. Comments and submissions are stored as separate datasets, split into monthly… See the full description on the dataset page: https://huggingface.co/datasets/open-index/arctic.hacker-news
Hacker News - Complete Archive
Every Hacker News item since 2006, live-updated every 5 minutes
What is it?
This dataset contains the complete Hacker News archive: every story, comment, Ask HN, Show HN, job posting, and poll ever submitted to the site. Hacker News is one of the longest-running and most influential technology communities on the internet, operated by Y Combinator since 2007. It has become the de facto gathering place for founders, engineers… See the full description on the dataset page: https://huggingface.co/datasets/open-index/hacker-news.IITB-IndicMonoDocIITB Document level Monolingual Corpora for Indian languages.
22 scheduled languages of India + English
(1) Assamese, (2) Bengali, (3) Gujarati, (4) Hindi, (5) Kannada, (6) Kashmiri, (7) Konkani, (8) Malayalam, (9) Manipuri, (10) Marathi, (11) Nepali, (12) Oriya, (13) Punjabi, (14) Sanskrit, (15) Sindhi, (16) Tamil, (17) Telugu, (18) Urdu (19) Bodo, (20) Santhali, (21) Maithili and (22) Dogri.
Language
Total (#Mil Tokens)
bn
5258.47
en
11986.53
gu
887.18
hi
11268.33
kn… See the full description on the dataset page: https://huggingface.co/datasets/cfilt/IITB-IndicMonoDoc.IndustryCorpus[中文主页]
Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.
To address these problems, we constructed and applied 22 industry data processing operators to… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus.host-index-testing-v2
Common Crawl Host Index v2
GitHub: https://github.com/commoncrawl/cc-host-index
Each crawl, we generate a Host Index, which aggregates information about each web hosted visited during the crawl. The
information is aggregated from the Common Crawl columnar index,
web graph, and raw crawler logs.
Quickstart
The dataset is Hive-partitioned on crawl (data/crawl=CC-MAIN-2025-18/*.parquet). Open the whole
dataset once, then filter with WHERE crawl = '...': because… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/host-index-testing-v2.IndustryCorpus_technology[中文主页]
Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.
To address these problems, we constructed and applied 22 industry data processing operators to… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus_technology.IndustryCorpus_finance[中文主页]
Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.
To address these problems, we constructed and applied 22 industry data processing operators to… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus_finance.IndustryCorpus_education[中文主页]
Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.
To address these problems, we constructed and applied 22 industry data processing operators to… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus_education.indic-align
IndicAlign
A diverse collection of Instruction and Toxic alignment datasets for 14 Indic Languages. The collection comprises of:
IndicAlign - Instruct
Indic-ShareLlama
Dolly-T
OpenAssistant-T
WikiHow
IndoWordNet
Anudesh
Wiki-Conv
Wiki-Chat
IndicAlign - Toxic
HHRLHF-T
Toxic-Matrix
We use IndicTrans2 (Gala et al., 2023) for the translation of the datasets.
We recommend the readers to check out our paper on Arxiv for detailed information on the curation process of these… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/indic-align.IndustryCorpus_news[中文主页]
Industry models play a crucial role in driving enterprise intelligence transformation and innovative development. High-quality industry data is key to improving the performance of large models and realizing industry applications. However, datasets currently used for industry model training generally suffer from issues such as insufficient data volume, low quality, and lack of domain expertise.
To address these problems, we constructed and applied 22 industry data processing operators to… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus_news.Nemotron-Personas-India
Nemotron-Personas-India
A compound AI approach to personas grounded in real-world distributions
वास्तविक दुनिया के वितरण पर आधारित व्यक्तित्वों के लिए एक मिश्रित AI दृष्टिकोण
Dataset Overview (डेटासेट अवलोकन)
Nemotron-Personas-India is an open-source (CC BY 4.0) dataset of synthetically-generated personas. This dataset is grounded in real-world demographic, geographic and personality trait distributions in India to capture the diversity and richness of… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-Personas-India.open-arxiv
Open arXiv
Every arXiv paper's metadata in one place: search, filter, and explore 40 years of science
What is it?
Open arXiv is the complete arXiv metadata dataset, covering titles, abstracts, authors, categories, DOIs, version history, and more. It is converted from the Cornell University Kaggle dataset into Parquet format for efficient querying and streaming.
The dataset contains 2.99M papers spanning from 1991 to 2026, packaged into 417 Parquet shards (Zstd… See the full description on the dataset page: https://huggingface.co/datasets/open-index/open-arxiv.Indian-Bank-Statements
Indian Bank Statement Synthetic Dataset
Synthetically generated Indian business bank statements with realistic transaction patterns, proper banking workflows, and India-specific features. Available in scanned PDF and digital JSON formats.
Scope: Current Accounts (business banking) only. Does not include personal/savings accounts.
Dataset Details
Curated by: AgamiAI Inc.
Language(s): English, Hindi (romanized)
License: Apache 2.0
Repository:… See the full description on the dataset page: https://huggingface.co/datasets/AgamiAI/Indian-Bank-Statements.Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1
Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1
Dataset Description:
Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1 is an RL dataset for training and evaluating a tool-using agent's ability to resist Indirect Prompt Injection (IPI) attacks hidden inside tool-returned environment data. In each record, the agent receives a benign user request that requires calling a read tool whose output contains an adversarial instruction disguised as legitimate domain content… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1.ccrawl-recrawl-domains
Common Crawl Domain Recrawl
Live fetches of the home page of every ranked domain in Common Crawl's web graph, rendered to Markdown as they are fetched
What is it?
Common Crawl's web graph ranks domains by how central they are, but it does not tell you what those domains actually serve today. This dataset walks that ranking from the top and fetches each domain's home page now, storing the response as one Parquet row with the body, the headers, the timing and the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-domains.vitco
ViTco
165,847,195 Vietnamese documents from 4 public corpora, 370.2 GB of Parquet, one schema
This dataset is the pinned public Vietnamese corpora as gao read them, every source put to one contract and one schema, before any cleaning.
Contents
What is it
What is in it
Where the text came from
How it is laid out
Reading it
What you can build with it
One row
The columns
What this repo is
What ships and what does not
Things to know before you use it
What this is… See the full description on the dataset page: https://huggingface.co/datasets/open-index/vitco.open-github
OpenGitHub
What is it?
This dataset contains every public event on GitHub: every push, pull request, issue, star, fork, code review, release, and discussion across all public repositories. GitHub is the world's largest software development platform, home to over 200 million repositories and the daily work of tens of millions of developers, from individual open-source contributors to the engineering teams behind the most widely used software on earth.
The archive currently… See the full description on the dataset page: https://huggingface.co/datasets/open-index/open-github.misconceptionsindic-hplt-v2
Indic HPLT v2
A multilingual pretraining corpus of 34,605,630 documents (~25.5B estimated tokens, ~218 GB raw JSONL) across 13 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data.
This is the larger successor to Indic HPLT v1 (9.8M docs, 11 languages). Compared to v1, this release adds 3 new Indic languages (Nepali, Odia, Assamese) and ~3.5× more documents overall.
Quick Start
from datasets import load_dataset
# Full training… See the full description on the dataset page: https://huggingface.co/datasets/ashtok897/indic-hplt-v2.IndicIFEval
IndicIFEval
Paper | GitHub
Instruction-following benchmarks remain predominantly English-centric,
leaving a critical evaluation gap for the hundreds of millions of Indic
language speakers. We introduce IndicIFEval, a benchmark evaluating constrained
generation of LLMs across 14 Indic languages using automatically verifiable,
rule-based instructions. It combines two complementary tracks: IndicIFEval-Trans,
translated prompts from IFEval carefully localized for Indic… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/IndicIFEval.indic-hplt-v1
Indic HPLT v1
A multilingual pretraining corpus of 9,836,075 documents (~8.4B estimated tokens) across 10 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data.
Quick Start
from datasets import load_dataset
# Full training split
ds = load_dataset("ashtok897/indic-hplt-v1", split="train")
# Filter by language
hi_ds = ds.filter(lambda x: x["lang"] == "hi")
# Streaming (recommended for large-scale use)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/ashtok897/indic-hplt-v1.IndMix
IndMix (https://arxiv.org/abs/2512.18834) is an Indonesian pretraining corpus built by combining six publicly available Indonesian datasets, applying Indonesian-specific quality filtering, and performing cross-dataset deduplication.
Subsets
Subset
Description
quality_filtered
Quality-filtered data before deduplication
minhash_deduped
Document-level MinHash deduplication
matched
Documents appearing in 2+ source datasets
The matched subset uses… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/IndMix.mc4-idA thoroughly cleaned version of the Italian portion of the multilingual
colossal, cleaned version of Common Crawl's web crawl corpus (mC4) by AllenAI.
Based on Common Crawl dataset: "https://commoncrawl.org".
This is the processed version of Google's mC4 dataset by AllenAI, with further cleaning
detailed in the repository README file.ccrawl-recrawl-urls
Common Crawl URL Recrawl
Live refetches of pages from Common Crawl's URL index, with the body inline and the text already extracted
What is it?
Common Crawl publishes which URLs it saw and when, but the page bodies live in WARC archives that are awkward to query and are as old as the crawl that made them. This dataset takes the URL index for a single monthly crawl and fetches the pages again now, storing each response as one Parquet row with the body, the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-urls.snowball-replay-index
Snowball replay index
This dataset is a compact membership and ordering index for an approximate replay of Snowball's 10,372,343,704,053-token
data store. It contains no source text or token arrays. The 6,301 Parquet files contain three columns:
source_id: logical source key; join it to the source_id field in sources.json
document_id: the retained XXH3-128 content hash as 16 bytes
bucket_id: domain_cluster * 5 + quality_bucket
Document join contract
document_id… See the full description on the dataset page: https://huggingface.co/datasets/open-athena/snowball-replay-index.t5gemma2-indonesia-instruct-v1
T5Gemma-2 Indonesian Instruct — Mono-Repo
Satu repositori dataset HF untuk seluruh data pelatihan T5-Gemma-2 bahasa Indonesia.
Diorganisasi per fungsi (fondasi → spesifik → preferensi) dengan folder/subfolder,
setiap config = folder dan berisi split train + validation (80:20) di level percakapan.
Struktur (by fungsi)
t5gemma2-indonesia-instruct-v1/
├── README.md
├── manifest.json
├── chat_idx_map.json
├── foundation/ ← FASE 1 · fondasi Bahasa… See the full description on the dataset page: https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-instruct-v1.vitco-clean
ViTco Clean
2,279,914 Vietnamese documents from 2 public corpora, 10.6 GB of Parquet, one schema
This dataset is the same corpora after the cleaning line: normalized, measured, filtered to Vietnamese prose, deduplicated on identity, and with the personal identifiers covered.
Contents
What is it
What is in it
Where the text came from
How it is laid out
Reading it
What you can build with it
One row
The columns
What this repo is
What ships and what does not
Things… See the full description on the dataset page: https://huggingface.co/datasets/open-index/vitco-clean.open-library
Open Library
The complete Open Library catalog in clean, analysis-ready Parquet. 150.0M+ records across 11 entity types, from ISBNs and author bios to reading logs and Wikidata links.
What is it?
Open Library is a complete snapshot of the Open Library database, an open project of the Internet Archive with the mission of creating "one web page for every book ever published." The catalog is community-edited and contains bibliographic records for millions of authors, works… See the full description on the dataset page: https://huggingface.co/datasets/open-index/open-library.Nemotron-Personas-India
Nemotron-Personas-India
A compound AI approach to personas grounded in real-world distributions
वास्तविक दुनिया के वितरण पर आधारित व्यक्तित्वों के लिए एक मिश्रित AI दृष्टिकोण
Dataset Overview (डेटासेट अवलोकन)
Nemotron-Personas-India is an open-source (CC BY 4.0) dataset of synthetically-generated personas. This dataset is grounded in real-world demographic, geographic and personality trait distributions in India to capture the diversity and… See the full description on the dataset page: https://huggingface.co/datasets/kshitijthakkar/Nemotron-Personas-India.IndicMMLU-Pro
IndicMMLU Dataset
This dataset contains the following languages:
punjabi
hindi
urdu
telugu
gujrati
kannada
tamil
marathi
bengali
UPLOAD
Cite our work.
This dataset is also described in IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding.
@dataset{kj2024indicmmlupro,
author = {Kj, Sankalp and Kumar, Ashutosh and Balaji, Laxmaan and Kotecha, Nikunj and Jain, Vinija and Chadha, Aman and Bhaduri, Sreyoshi},
title =… See the full description on the dataset page: https://huggingface.co/datasets/LinguaLift/IndicMMLU-Pro.