datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
host-index-testing-v2
Common Crawl Host Index v2
GitHub: https://github.com/commoncrawl/cc-host-index
Each crawl, we generate a Host Index, which aggregates information about each web hosted visited during the crawl. The
information is aggregated from the Common Crawl columnar index,
web graph, and raw crawler logs.
Quickstart
The dataset is Hive-partitioned on crawl (data/crawl=CC-MAIN-2025-18/*.parquet). Open the whole
dataset once, then filter with WHERE crawl = '...': because… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/host-index-testing-v2.indic-align
IndicAlign
A diverse collection of Instruction and Toxic alignment datasets for 14 Indic Languages. The collection comprises of:
IndicAlign - Instruct
Indic-ShareLlama
Dolly-T
OpenAssistant-T
WikiHow
IndoWordNet
Anudesh
Wiki-Conv
Wiki-Chat
IndicAlign - Toxic
HHRLHF-T
Toxic-Matrix
We use IndicTrans2 (Gala et al., 2023) for the translation of the datasets.
We recommend the readers to check out our paper on Arxiv for detailed information on the curation process of these… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/indic-align.ccrawl-recrawl-domains
Common Crawl Domain Recrawl
Live fetches of the home page of every ranked domain in Common Crawl's web graph, rendered to Markdown as they are fetched
What is it?
Common Crawl's web graph ranks domains by how central they are, but it does not tell you what those domains actually serve today. This dataset walks that ranking from the top and fetches each domain's home page now, storing the response as one Parquet row with the body, the headers, the timing and the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-domains.open-github
OpenGitHub
What is it?
This dataset contains every public event on GitHub: every push, pull request, issue, star, fork, code review, release, and discussion across all public repositories. GitHub is the world's largest software development platform, home to over 200 million repositories and the daily work of tens of millions of developers, from individual open-source contributors to the engineering teams behind the most widely used software on earth.
The archive currently… See the full description on the dataset page: https://huggingface.co/datasets/open-index/open-github.indic-hplt-v2
Indic HPLT v2
A multilingual pretraining corpus of 34,605,630 documents (~25.5B estimated tokens, ~218 GB raw JSONL) across 13 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data.
This is the larger successor to Indic HPLT v1 (9.8M docs, 11 languages). Compared to v1, this release adds 3 new Indic languages (Nepali, Odia, Assamese) and ~3.5× more documents overall.
Quick Start
from datasets import load_dataset
# Full training… See the full description on the dataset page: https://huggingface.co/datasets/ashtok897/indic-hplt-v2.indic-hplt-v1
Indic HPLT v1
A multilingual pretraining corpus of 9,836,075 documents (~8.4B estimated tokens) across 10 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data.
Quick Start
from datasets import load_dataset
# Full training split
ds = load_dataset("ashtok897/indic-hplt-v1", split="train")
# Filter by language
hi_ds = ds.filter(lambda x: x["lang"] == "hi")
# Streaming (recommended for large-scale use)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/ashtok897/indic-hplt-v1.ccrawl-recrawl-urls
Common Crawl URL Recrawl
Live refetches of pages from Common Crawl's URL index, with the body inline and the text already extracted
What is it?
Common Crawl publishes which URLs it saw and when, but the page bodies live in WARC archives that are awkward to query and are as old as the crawl that made them. This dataset takes the URL index for a single monthly crawl and fetches the pages again now, storing each response as one Parquet row with the body, the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-urls.snowball-replay-index
Snowball replay index
This dataset is a compact membership and ordering index for an approximate replay of Snowball's 10,372,343,704,053-token
data store. It contains no source text or token arrays. The 6,301 Parquet files contain three columns:
source_id: logical source key; join it to the source_id field in sources.json
document_id: the retained XXH3-128 content hash as 16 bytes
bucket_id: domain_cluster * 5 + quality_bucket
Document join contract
document_id… See the full description on the dataset page: https://huggingface.co/datasets/open-athena/snowball-replay-index.t5gemma2-indonesia-instruct-v1
T5Gemma-2 Indonesian Instruct — Mono-Repo
Satu repositori dataset HF untuk seluruh data pelatihan T5-Gemma-2 bahasa Indonesia.
Diorganisasi per fungsi (fondasi → spesifik → preferensi) dengan folder/subfolder,
setiap config = folder dan berisi split train + validation (80:20) di level percakapan.
Struktur (by fungsi)
t5gemma2-indonesia-instruct-v1/
├── README.md
├── manifest.json
├── chat_idx_map.json
├── foundation/ ← FASE 1 · fondasi Bahasa… See the full description on the dataset page: https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-instruct-v1.open-library
Open Library
The complete Open Library catalog in clean, analysis-ready Parquet. 150.0M+ records across 11 entity types, from ISBNs and author bios to reading logs and Wikidata links.
What is it?
Open Library is a complete snapshot of the Open Library database, an open project of the Internet Archive with the mission of creating "one web page for every book ever published." The catalog is community-edited and contains bibliographic records for millions of authors, works… See the full description on the dataset page: https://huggingface.co/datasets/open-index/open-library.IndicMMLU-Pro
IndicMMLU Dataset
This dataset contains the following languages:
punjabi
hindi
urdu
telugu
gujrati
kannada
tamil
marathi
bengali
UPLOAD
Cite our work.
This dataset is also described in IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding.
@dataset{kj2024indicmmlupro,
author = {Kj, Sankalp and Kumar, Ashutosh and Balaji, Laxmaan and Kotecha, Nikunj and Jain, Vinija and Chadha, Aman and Bhaduri, Sreyoshi},
title =… See the full description on the dataset page: https://huggingface.co/datasets/LinguaLift/IndicMMLU-Pro.indian-legal-documents
Indian Legal Documents
Open Indian statutory and legal-document data for AI, search, and legal research.
This dataset is part of the KanoonGPT Open Legal Data Initiative — an effort to make Indian legal data easier to access, structure, and build on for open-source research, legal tech, and production AI systems.
KanoonGPT is building structured Indian legal datasets and data infrastructure for open-source, research, and enterprise AI applications. Learn more at kanoongpt.in.… See the full description on the dataset page: https://huggingface.co/datasets/KanoonGPT/indian-legal-documents.industrial-instruction-dataset
Industrial-Instruction Dataset
Industrial-Instruction provides benchmark and training-ready QA instances derived from industrial technical reports, designed to evaluate robustness under realistic retrieval conditions. Samples are grounded in retrieved evidence and include irrelevant retrieval, single-/multi-document support, and single-/multi-document answer settings.
Paper
Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and… See the full description on the dataset page: https://huggingface.co/datasets/Parssky/industrial-instruction-dataset.indian-court-decisions
Indian Court Decisions
A large-scale dataset of Indian court decisions with full text, metadata, and outcome labels covering the Supreme Court of India and 25 High Courts (1950–2026).
Dataset Summary
Config
Train
Validation
Test
Total
high_courts
11,682,776
1,459,319
1,457,934
14,600,029
supreme_court
40,044
4,990
5,019
50,053
Total
14,650,082
This is one of the largest publicly available legal NLP datasets, containing over 14.6 million… See the full description on the dataset page: https://huggingface.co/datasets/overthelex/indian-court-decisions.index-souverainete
Index Souveraineté Le Souv
Le dataset public de référence sur les entreprises françaises stratégiques cédées à des capitaux étrangers, et sur les entreprises souveraines à capitaux français.
Source canonique : Le Souv — média indépendant consacré à la souveraineté économique et politique française.
URL canonique : https://lesouv.fr/index-souverainete.json
Licence : Creative Commons BY 4.0 (réutilisation libre avec attribution à Le Souv).
Mise à jour : continue, regénération… See the full description on the dataset page: https://huggingface.co/datasets/LeSouv/index-souverainete.bhasha-wiki-indic
Bhasha Wiki Indic
This dataset has Wikipedia articles pertaining to Indian context.
Dataset Details
Dataset Description
The dataset is built from Wikipedia articles taken from wikimedia/wikipedia.
We filtered, cleaned and translated English articles related to India and Indian context out of entire dataset.
Each example has contents of a full cleaned wikipedia article and it's translations in 6 Indian languages.
Curated by: Soket AI Labs
Language(s) (NLP):… See the full description on the dataset page: https://huggingface.co/datasets/soketlabs/bhasha-wiki-indic.indian-court-decisions
Indian Court Decisions
A large-scale dataset of Indian court decisions with full text, metadata, and outcome labels covering the Supreme Court of India and 25 High Courts (1950–2026).
Dataset Summary
Config
Train
Validation
Test
Total
high_courts
11,682,776
1,459,319
1,457,934
14,600,029
supreme_court
40,044
4,990
5,019
50,053
Total
14,650,082
This is one of the largest publicly available legal NLP datasets, containing over 14.6 million… See the full description on the dataset page: https://huggingface.co/datasets/rtarun789/indian-court-decisions.morocco-public-tenders-open-index
🇲🇦 Morocco Public Procurement & Tenders Open Index (2020–2026)
Official Open Data repository maintained by Soumit Intelligence EngineFree for public, educational, academic, research, and non-commercial use under CC-BY-NC 4.0.
⚖️ Non-Commercial License Terms (CC-BY-NC 4.0)
This dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC 4.0) license.
✅ Free to use & share: For research, education, data science… See the full description on the dataset page: https://huggingface.co/datasets/abdnaouri/morocco-public-tenders-open-index.tomo-traces
Tomo Agent Traces
Every tomo-labs run, published as it happens: the full agent trace, plus the boards and cost analyses regenerated from every result on each commit.
What is it?
This dataset is the running record of tomo-labs, the agent-evaluation harness for tomo and the coding agents it is measured against.
Every time the harness runs a tool on a scenario, it captures the whole conversation the agent had with the model, converts it to the Hub's agent-trace… See the full description on the dataset page: https://huggingface.co/datasets/open-index/tomo-traces.inductive-forecasting-data
Inductive Forecasting Study — Anonymous Data Release
This repository is the anonymous data companion to a paper studying behavioral
signatures of inductive reasoning in language-model forecasts. It packages the
frozen inputs, model responses, row-level scores, and aggregate result artifacts
used by the paper's four main experiments, together with synthetic appendix
transfer studies.
The release is organized as Hugging Face dataset configurations so each study can
be loaded… See the full description on the dataset page: https://huggingface.co/datasets/od2961/inductive-forecasting-data.IndustryInstruction-Chinese
中文行业指令数据集
💻 Github Repo
简介
本数据集提取了原数据集 BAAI/IndustryInstruction 中源语言为中文的部分，并做了清洗。数据集分为单轮对话和多轮对话两个子集。
本数据集包含的行业及具体数据如下：
领域
单轮对话数目
多轮对话数目
AeroSpace
72667
0
Artificial-Intelligence
43906
0
Automobiles
78036
0
Finance-Economics
40135
0
Health-Medicine
177152
105320
Hospitality-Catering
39261
0
Law-Justice
43485
0
Literature-Emotions
44841
0
Subject-Education
271402
73
Technology-Research
41751
0
Transportation
51505
0
Travel-Geography
37150… See the full description on the dataset page: https://huggingface.co/datasets/Mxode/IndustryInstruction-Chinese.indian-responsible-ai-benchmark
Indian Responsible AI Benchmark
A comprehensive benchmark for evaluating responsible AI behavior in Indian contexts — covering 212 adversarial and safety-critical prompts across 22 evaluation categories, 10 Indian language regions, and 8 Responsible AI dimensions.
Why This Benchmark?
Most AI safety benchmarks are US/Western-centric. Indian users face unique challenges:
Caste dynamics not captured by Western bias benchmarks
India/US context confusion (models… See the full description on the dataset page: https://huggingface.co/datasets/responsible-ai-labs/indian-responsible-ai-benchmark.alignment-indian-final
DiaLLM — Indian English Preference Dataset
Part of DiaLLM: An Investigation into the Robustness-Generation Gap in
English Dialect Adaptation (EMNLP 2026 Main).
18,402 preference pairs for Indian English (en-IN), used for explicit-thread
DPO/GRPO/GSPO training targeting this variety.
Construction
Built from the UltraFeedback preference dataset (Cui et al., 2023):
the originally-preferred completion is transformed into a dialectal variant
using Multi-VALUE (Ziems… See the full description on the dataset page: https://huggingface.co/datasets/surrey-nlp/alignment-indian-final.AgentWorldBench-Terminal-V2
AgentWorldBench-Terminal-V2
AgentWorldBench-Terminal-V2 is our improved subset of the terminal split from
Qwen/AgentWorldBench
(Zou et al., 2026). Given the history of a Linux terminal session,
the model is evaluated on its ability to predict the output of the next command.
In the original AgentWorldBench, some samples have ground-truth outputs that depend on environment
details missing from the session history. Since the sessions are based on Terminal-Bench environments,
the… See the full description on the dataset page: https://huggingface.co/datasets/inductionlabs/AgentWorldBench-Terminal-V2.IndianPersona-1M
IndianPersona-1M — Synthetic Indian Demographics & LLM Agent Personas
1,000,000 culturally-grounded synthetic Indian demographic profiles plus
250,000 ready-to-use LLM agent personas — generated entirely with the open-source
indian-fakedata library
(PyPI · npm).
This dataset is 100% synthetic. Every row carries synthetic = true.
All identifiers (Aadhaar, PAN, voter ID, phone, email) are fabricated and exist in no
government or commercial database. No real individual is… See the full description on the dataset page: https://huggingface.co/datasets/Abhay557/IndianPersona-1M.roofing-cost-index
US Residential Roofing Cost Index (2026)
Dataset Summary
This dataset contains highly localized, objective residential roof replacement pricing indices for 505 major US cities across all 50 states. All pricing figures represent synthesized, algorithmically compiled estimates for the year 2026 by the Shingle Geek pricing engine.
The dataset provides dual cost models to inject complete transparency into the residential home improvement market:
Fair Contractor… See the full description on the dataset page: https://huggingface.co/datasets/ShingleGeek/roofing-cost-index.indonesian-recipes
Resep Masakan Indonesia 🍛
Kumpulan resep masakan Indonesia autentik — dari rendang sampai es cendol, lengkap dengan bahan, langkah, tingkat kesulitan, waktu, dan daerah asal.
Kenapa dataset ini ada?
Resep adalah salah satu konten paling dicari untuk LLM (assistant masak) — tapi dataset resep Indonesia di HF nyaris kosong (cuma 1 yang 34 likes). Gw isi gap itu dengan resep-resep yang benar-benar asli Indonesia, bukan versi western yang diterjemahkan.… See the full description on the dataset page: https://huggingface.co/datasets/LorthGyu/indonesian-recipes.t5gemma2-indonesia-chat-formatted
T5Gemma-2 Indonesian Chat & QA Dataset
A high-quality Indonesian language multi-turn conversation and reading comprehension dataset, specifically formatted for instruction tuning of sequence-to-sequence (Seq2Seq) models like T5-Gemma / T5-Gemma-2.
Dataset Description
This dataset contains over 7,400 multi-turn conversations and document-based Q&A in Bahasa Indonesia. It covers diverse topics including everyday life, technology, general knowledge, and structured… See the full description on the dataset page: https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-chat-formatted.indo-bloom-corpus
🇮🇩 Indo-Bloom-AQG: A Unified Framework for Controllable Indonesian AQG
⚠️ RESEARCH ARTIFACT STATUS: SILVER VERSION (Work in Progress)
This dataset serves as the preliminary corpus (Silver Standard) for the ongoing Doctoral Dissertation at Universitas Negeri Malang (UM).
Current State: Unannotated / Pre-validation with Heuristic Bloom Labels
Target Final State: Gold Standard (Expert Validated with Bloom's Taxonomy Labels)
🔒 FROZEN — v0.1 Silver
This version is permanently… See the full description on the dataset page: https://huggingface.co/datasets/Firmansyah-Ibrahim/indo-bloom-corpus.K-Paths-inductive-reasoning-drugbank
🔗 This dataset is part of the study:
K-Paths: Reasoning over Graph Paths for Drug Repurposing and Drug Interaction Prediction
📖 Read the Paper
💾 GitHub Repository
DrugBank: Inductive Reasoning Dataset
This dataset contains drug pairs annotated with 86 pharmacological relationships (e.g.,DrugA may increase the anticholinergic activities of DrugB).
Each entry includes two drugs, an interaction label, drug descriptions, and structured/natural language representations… See the full description on the dataset page: https://huggingface.co/datasets/Tassy24/K-Paths-inductive-reasoning-drugbank.