datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
4chan-datasetsPlease see repo to turn the text file into json/csv format
Deleted some boards, since they are already archived by https://archive.4plebs.org/
llm_datasetsMixed-Arabic-Datasets-Repo
Dataset Card for "Mixed Arabic Datasets (MAD) Corpus"
The Mixed Arabic Datasets Corpus : A Community-Driven Collection of Diverse Arabic Texts
Dataset Description
The Mixed Arabic Datasets (MAD) presents a dynamic compilation of diverse Arabic texts sourced from various online platforms and datasets. It addresses a critical challenge faced by researchers, linguists, and language enthusiasts: the fragmentation of Arabic language datasets across the Internet. With MAD, we… See the full description on the dataset page: https://huggingface.co/datasets/M-A-D/Mixed-Arabic-Datasets-Repo.YuLan-Mini-Text-Datasets
News
[2025.04.11] Add dataset mixture: link.
[2025.03.30] Text datasets upload finished.
This is text dataset.
这是文本格式的数据集。
Since we have used BPE-Dropout, in order to ensure accuracy, you can find the tokenized dataset here.
由于我们使用了BPE-Dropout，为了保证准确性，你可以在这里找到分词后的数据。
For more information, please refer to our datasets details and preprocess details.
Contributing
We welcome any form of contribution, including feedback on model bad cases, feature suggestions, and example… See the full description on the dataset page: https://huggingface.co/datasets/yulan-team/YuLan-Mini-Text-Datasets.wiki_snippets
Dataset Card for "wiki_snippets"
Dataset Summary
Wikipedia version split into plain text snippets for dense semantic indexing.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
We show detailed information for 2 configurations of the dataset (with 100 snippet passage length and 0 overlap) in
English:
wiki40b_en_100_0: Wiki-40B
wikipedia_en_100_0: Wikipedia
Data Instances… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/wiki_snippets.glucose
Dataset Card for [Dataset Name]
Dataset Summary
GLUCOSE: GeneraLized and COntextualized Story Explanations, is a novel conceptual framework and dataset for commonsense reasoning. Given a short story and a sentence X in the story, GLUCOSE captures ten dimensions of causal explanation related to X. These dimensions, inspired by human cognitive psychology, cover often-implicit causes and effects of X, including events, location, possession, and other attributes.… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/glucose.speculators-ci-datasets
speculator-tutorial
Raw vs. on-policy regenerated conversation data for training speculative-decoding
drafters (EAGLE-3 / DFlash / DSpark style), with the original source data kept alongside
so you can see exactly what regeneration changes and why it matters.
Prompts come from UltraChat-200k. The verifier / teacher model is Qwen/Qwen3-8B.
Why regenerate at all?
A speculative-decoding drafter is trained to predict what the verifier would say next.
If you train it… See the full description on the dataset page: https://huggingface.co/datasets/inference-optimization/speculators-ci-datasets.fractus-datasets
Fractus Datasets — the neuroscience-grounded training corpus
A proprietary, neuroscience-derived training corpus for the Fractus Continuous Thought Engine — ~3–4B tokens mapping real brain mechanisms to software/AI architecture, plus cognitive skills, code, esoteric tradition, and lexical knowledge.
Curator: Philippe-Antoine Robert · rpa.tu@proton.me · 2026
What this dataset collection IS
Fractus is a non-transformer Continuous Cognitive Agent whose architecture… See the full description on the dataset page: https://huggingface.co/datasets/thefinalboss/fractus-datasets.verified-defi-datasets
Verified Solana Sealevel & Anchor Program Optimization Fine-Tuning Corpus
Dataset Description
High-density, verified AI fine-tuning dataset in ALPACA format.
Domain: Solana Sealevel & Anchor Program Optimization
Verified Records: 3
Estimated Tokens: 339
Quality QA Score: 99.0%
Monetization Status: Direct Zero-Gas Web3 & HuggingFace Distribution
MMLA-Datasets
Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
1. Introduction
MMLA is the first comprehensive multimodal language analysis benchmark for evaluating foundation models. It has the following features:
Large Scale: 61K+ multimodal samples.
Various Sources: 9 datasets.
Three Modalities: text, video, and audio
Both Acting and Real-world Scenarios: films, TV series, YouTube, Vimeo, Bilibili, TED, improvised scripts, etc.
Six Core… See the full description on the dataset page: https://huggingface.co/datasets/THUIAR/MMLA-Datasets.chess_datasets
Dataset Card for Dataset Name
This dataset card aims to be a base template for new datasets. It has been generated using this raw template.
Dataset Details
Dataset Description
Curated by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Dataset Sources [optional]
Repository: [More… See the full description on the dataset page: https://huggingface.co/datasets/p11-p11/chess_datasets.id_newspapers_2018
Dataset Card for Indonesian Newspapers 2018
Dataset Summary
The dataset contains around 500K articles (136M of words) from 7 Indonesian newspapers: Detik, Kompas, Tempo,
CNN Indonesia, Sindo, Republika and Poskota. The articles are dated between 1st January 2018 and 20th August 2018
(with few exceptions dated earlier). The size of uncompressed 500K json files (newspapers-json.tgz) is around 2.2GB,
and the cleaned uncompressed in a big text file (newspapers.txt.gz) is… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/id_newspapers_2018.cpt_instruction_datasets
Instruction datasets
Collection of synthetic instruction datasets used during the continued pretraining of Model-small-instr-1, Model-small-instr-2 and Model-small-instr-3. You can currently find these models under: Llama-3.1-Carballo-Instr1 and Llama-3.1-Carballo-Instr3.
Dataset creation
Datasets were created using two different techniques:
Adapting already existing datasets or corpora by modifying their format to make them suitable for including instructions during… See the full description on the dataset page: https://huggingface.co/datasets/proxectonos/cpt_instruction_datasets.Mixed-Arabic-Datasets-Repo
Dataset Card for "Mixed Arabic Datasets (MAD) Corpus"
The Mixed Arabic Datasets Corpus : A Community-Driven Collection of Diverse Arabic Texts
Dataset Description
The Mixed Arabic Datasets (MAD) presents a dynamic compilation of diverse Arabic texts sourced from various online platforms and datasets. It addresses a critical challenge faced by researchers, linguists, and language enthusiasts: the fragmentation of Arabic language datasets across the Internet. With… See the full description on the dataset page: https://huggingface.co/datasets/yrrhall/Mixed-Arabic-Datasets-Repo.ToolGen-Datasets
How to use?
Before making use of this dataset, you may need to add the tokens to the vocabulary. For HuggingFace transformers tokenizer, the following is an example code snippet to add tokens.
from unidecode import unidecode
import transformers
with open('virtual_tokens.txt', 'r') as f:
virtual_tokens = f.readlines()
virtual_tokens = [unidecode(vt.strip()) for vt in virtual_tokens]
model_name_or_path = "meta-llama/Meta-Llama-3-8B"
# Load tokenizer and add tokens into… See the full description on the dataset page: https://huggingface.co/datasets/reasonwang/ToolGen-Datasets.to-tool-call-datasets
🛠️ To-Tool-Call Datasets
A unified Qwen3-style tool-call corpus for SFT, GRPO, and agent training
To-Tool-Call Datasets is a curated mirror of public tool-call and function-calling corpora, re-serialized into one training-ready messages JSONL convention.
Quick Start ·
At a Glance ·
Format ·
Sources ·
Training Notes
[!IMPORTANT]
This repository is a format-harmonization layer, not a new claim of ownership over the… See the full description on the dataset page: https://huggingface.co/datasets/zhangdw/to-tool-call-datasets.cawac
Dataset Card for caWaC
Dataset Summary
caWaC is a 780-million-token web corpus of Catalan built from the .cat top-level-domain in late 2013.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
Dataset is monolingual in Catalan language.
Dataset Structure
Data Instances
[More Information Needed]
Data Fields
[More Information Needed]
Data Splits
[More Information Needed]
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/cawac.repairllama-datasets
RepairLLaMA - Datasets
Contains the processed fine-tuning datasets for RepairLLaMA.
Instructions to explore the dataset
To load the dataset, you must define which revision (i.e., which input/output representation pair) you want to load.
from datasets import load_dataset
# Load ir1xor1
dataset = load_dataset("ASSERT-KTH/repairllama-datasets", "ir1xor1")
# Load irXxorY
dataset = load_dataset("ASSERT-KTH/repairllama-datasets", "irXxorY")
Citation
If you use… See the full description on the dataset page: https://huggingface.co/datasets/ASSERT-KTH/repairllama-datasets.combined-chat-datasets
Combined Chat Datasets
A standardized, unified collection of 30 conversational AI datasets -- spanning organic in-the-wild chats, voluntary sharing, side-by-side preferences, conversation trees, RLHF pairs, and crowdsourced instruction tuning data -- normalized to a single schema for easy joint use.
This dataset is a re-distribution. It does not relicense the underlying data.
See the Legal & Licensing section -- you must comply with each source dataset's original license.… See the full description on the dataset page: https://huggingface.co/datasets/viktor-shcherb/combined-chat-datasets.consolidated-datasets
Dataset Card for YouTubeTranscriptData
Dataset Details
Dataset Description
This dataset contains transcripts of around 167K youtube videos that include coding lectures, podcasts, interviews, news videos, commentary and song lyrics. Also there are multiple files that have been generated using webscrapping.
Curated by: Shivendra Singh
License: [none]
Dataset Sources
Repository: SmallLanguageModel
Demo [optional]: [More Information Needed]… See the full description on the dataset page: https://huggingface.co/datasets/shivendrra/consolidated-datasets.farsi_news
Dataset Card for FarsiNews
Dataset Summary
https://github.com/sci2lab/Farsi-datasets
Contains Farsi (Persian) datasets for Machine Learning tasks, particularly NLP.
These datasets have been extracted from the RSS feed of two Farsi news agency websites:
Hamshahri
RadioFarda
Supported Tasks and Leaderboards
[More Information Needed]
Languages
[More Information Needed]
Dataset Structure
[More Information Needed]
Data Instances… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/farsi_news.tashkeela
Dataset Card for Tashkeela
Dataset Summary
It contains 75 million of fully vocalized words mainly
97 books from classical and modern Arabic language.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
The dataset is based on Arabic.
Dataset Structure
Data Instances
{'book':… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/tashkeela.cs_restaurants
Dataset Card for Czech Restaurant
Dataset Summary
This is a dataset for NLG in task-oriented spoken dialogue systems with Czech as the target language. It originated as a translation of the English San Francisco Restaurants dataset by Wen et al. (2015). The domain is restaurant information in Prague, with random/fictional values. It includes input dialogue acts and the corresponding outputs in Czech.
Supported Tasks and Leaderboards
other-intent-to-text:… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/cs_restaurants.lma_datasets
LMA Phase 1 --- Hindi and Nepali pretraining corpora
Two monolingual corpora built for a pair of ~25M-parameter decoder-only
Transformers. Hindi is the higher-resource language, Nepali the lower-resource
one. Both are written in Devanagari (U+0900-U+097F), so script cannot be used
to tell them apart --- separating them is the central technical problem this
dataset solves rather than assumes.
language
documents
characters
manual (chars)
tokens
manual (tokens)
train
val
test… See the full description on the dataset page: https://huggingface.co/datasets/meet5568/lma_datasets.youtube_caption_corrections
Dataset Card for YouTube Caption Corrections
Dataset Summary
This dataset is built from pairs of YouTube captions where both an auto-generated and a manually-corrected caption are available for a single specified language. It currently only in English, but scripts at repo support other languages. The motivation for creating it was from viewing errors in auto-generated captions at a recent virtual conference, with the hope that there could be some way to help correct those… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/youtube_caption_corrections.tis-subset-datasets-Llama-2-7b-hf
Targeted Instruction Selection Subsets (Llama-2-7b-hf)
This repository contains pre-computed instruction training subsets selected from a large candidate pool for targeted instruction fine-tuning, as presented in the paper A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't).
Paper: https://huggingface.co/papers/2602.14696
GitHub Repository: https://github.com/dcml-lab/targeted-instruction-selection
Description
Instruction… See the full description on the dataset page: https://huggingface.co/datasets/Harvard-DCML/tis-subset-datasets-Llama-2-7b-hf.airecon-datasets
AIRecon Security Datasets
Curated security knowledge datasets for AIRecon — an AI-powered security reconnaissance tool that runs 100% locally with Ollama.
These datasets augment the LLM agent's knowledge for penetration testing, reconnaissance, vulnerability analysis, and security research workflows.
How AIRecon Uses These Datasets
Dataset → Phase Mapping
Dataset
Primary Phase
What It Provides
recon-playbook
RECON
Agent methodology, phase tactics… See the full description on the dataset page: https://huggingface.co/datasets/pikpikcu/airecon-datasets.REP-datasets
REP-datasets — reasoning traces exposed by REP, used to train the released students
Released with "Hidden Thoughts Are Not Secret: Reasoning-Trace Exposure in
LLMs" (EMNLP 2026, arXiv:2606.00642). Code: https://github.com/0x-yuan/REP ·
Models: Chia-Mu-Lab/REP-models.
Each config is the exact training set of one released student. Questions come
from OpenThoughts-114k (math); traces were exposed from open-weight victims
(Qwen3-14B / Qwen3-32B) with the REP prompt and are the… See the full description on the dataset page: https://huggingface.co/datasets/Chia-Mu-Lab/REP-datasets.hebrew_this_world
Dataset Card for HebrewSentiment
Dataset Summary
HebrewThisWorld is a data set consists of 2028 issues of the newspaper 'This World' edited by Uri Avnery and were published between 1950 and 1989. Released under the AGPLv3 license.
Data Annotation:
Supported Tasks and Leaderboards
Language modeling
Languages
Hebrew
Dataset Structure
csv file with "," delimeter
Data Instances
Sample:
{
"issue_num": 637,
"page_count": 16… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/hebrew_this_world.ZamAi-Pashto-Datasets-V2
ZamAI Cleaned Pashto Dataset V2
Languages: psLicense: apache-2.0Task categories: summarization, text-generation, feature-extractionSize categories: 10K<n<100K
Summary
This dataset is part of the ZamAI Pashto data collection. It is intended for summarization, text-generation, feature-extraction tasks in Pashto.
How to use
from datasets import load_dataset
dataset = load_dataset("tasal9/ZamAi-Pashto-Datasets-V2")
print(dataset)
Configs… See the full description on the dataset page: https://huggingface.co/datasets/tasal9/ZamAi-Pashto-Datasets-V2.