datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SystemCheck
Dataset Card for SystemCheck
Dataset Summary
[Project Repo] [🏁 Checkpoints]
This repository contains data for our paper, SystemCheck: A Closer Look at System Prompt Reliability, which studies the reliability of system prompts in large language models.
SystemCheck is a collection of LLM training and evaluation datasets designed to study the robustness of LLM guardrails. It contains a set of 3000+ system prompts scraped from the ChatGPT store and HuggingChat, SFT/DPO… See the full description on the dataset page: https://huggingface.co/datasets/normster/SystemCheck.NormAd
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
The NormAd dataset is from the paper "NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models".
Code at GitHub Repo.
Data Update (July 27, 2026):
We've fixed some inconsistencies in the dataset and updated the data file. If you've downloaded the dataset previously, please re-download.
Dataset Description
NormAd-Eti is a benchmark… See the full description on the dataset page: https://huggingface.co/datasets/akhilayerukola/NormAd.Multilingual-Normalizer
Multilingual TTS text normalizer (written → spoken)
Training pairs for fine-tuning a small LLM as a text-to-speech normalizer: text is a sentence
the way people type it (digits, currency symbols, dates, phone numbers, …) and normalized is the
exact spoken form, in the same language, with nothing left that a TTS model cannot say.
52,698 rows — 16 monolingual locales and 6 Malaysian code-switched pairs. Every row is
digit-free on the spoken side.
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/Scicom-intl/Multilingual-Normalizer.task093_conala_normalize_lists
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task093_conala_normalize_lists
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks}… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task093_conala_normalize_lists.yoruba-normalization-pairs
Normalization pairs dataset
What this is
24,475 pairs of Yorùbá text, each a corrupted form next to its canonical form, labelled by corruption type. I built it for testing orthographic normalization code.
The library
This dataset was built alongside yotext, a Python library for Yorùbá orthographic normalization and diacritic restoration. The library is on PyPI at https://pypi.org/project/yotext/ and the source is at… See the full description on the dataset page: https://huggingface.co/datasets/adedejimakinde/yoruba-normalization-pairs.calib-agentic-normalized
calib-agentic (v1)
Normalized calibration corpus for post-training quantization. Every row is OpenAI-shaped
(messages + tools) regardless of upstream dialect, with provenance in origin and
precomputed stats for stratified selection.
Schema
messages[] — role (system/user/assistant/tool), content, name, tool_call_id,
tool_calls[] (name, arguments as a JSON string), reasoning_content
tools[] — name, description, parameters (JSON string of the JSON-Schema)
origin —… See the full description on the dataset page: https://huggingface.co/datasets/digi-texx/calib-agentic-normalized.sp-sft-normal-300k
model-raising-pbsft-instruct-300k
A constitution-aware paired SFT dataset of 300,000 general-purpose (WildChat) instruct
prompts. Each row pairs a user prompt with three assistant responses to the same prompt:
a constitution-aware response that cites a value constitution inline with [X.Y] markers,
a constitution-invisible rendering of that same response (no markers, no constitution vocabulary), and
the original response that shipped with the prompt in WildChat-1M.
It is part… See the full description on the dataset page: https://huggingface.co/datasets/dlab-spp/sp-sft-normal-300k.Mathlib-Normalized-Sexpr
Mathlib Normalized S-Expressions
Lean 4 proof states from Mathlib, paired with the tactic applied at each
step, in three representations extracted directly from the Lean kernel:
Source-faithful S-expressions of the goal and every hypothesis, as
Lean elaborated them.
Normalized S-expressions of the same state, with stable local-context
indices suitable for model input.
Annotated tactic syntax -- the original tactic's syntax tree with
identifier leaves resolved to the constants… See the full description on the dataset page: https://huggingface.co/datasets/jajostrains/Mathlib-Normalized-Sexpr.Korean-OpenThoughts-114k-NormalizedKorean-OpenThoughts-114k-Normalized
상세
데이터셋 설명
OpenThoughts-114k-Normalized 데이터셋의 한국어 번역본입니다.
OpenAI gpt-4o-mini를 통해 번역됐습니다.
Shared by llami-team
Language(s) (NLP): Korean
Uses
한국어 reasoning 모델 distillation
reasoning cold-start 데이터셋
Dataset Structure
question: 질문
reasoning: 추론 과정
response: 응답
Dataset Creation
[LLAMI Team] (https://llami.net)
LLAMI Github
lemon-mint
Source Data
OpenThoughts-114k-Normalized
router-chat-normalized-1m
Router Chat Normalized 1M
Dataset Description
Router Chat Normalized 1M is a multilingual conversational dataset containing 1,353,300 conversations normalized from multiple public chat datasets with automatic language detection.
Dataset Structure
The dataset contains 2 split(s): train, test.
Each conversation includes: conversation_id, messages (list of {role, content} structs), source dataset, detected language, and language confidence score.
Source… See the full description on the dataset page: https://huggingface.co/datasets/whoisandy/router-chat-normalized-1m.uk-text-normalization
Український TTS-нормалізатор — датасет
Пари «письмовий текст → як його вимовляють» для української. Числа, дати, час,
гроші, одиниці, скорочення, коди, телефони, IBAN, домени, пошта, римські
цифри, латинські вкраплення — те, що треба розгорнути словами перед синтезом
мовлення.
{"task_id": 0,
"combo_names": ["Кількісні числівники (написані цифрами)",
"Порядкові числівники (написані цифрами з закінченням)"],
"original": "На 1-й полиці стоять 4 книги."… See the full description on the dataset page: https://huggingface.co/datasets/skypro1111/uk-text-normalization.normalisation-S2S-training
Tibetan Normalisation - S2S Training Data
A large-scale parallel training dataset for Tibetan text normalisation, containing approximately 2 million line pairs mapping diplomatic (non-standard, abbreviated) Tibetan manuscript text to Standard Classical Tibetan. This dataset was used to train the sequence-to-sequence normalisation models (tokenised S2S model and non-tokenised S2S model) released as part of the PaganTibet project.
The dataset combines a manually curated gold-standard… See the full description on the dataset page: https://huggingface.co/datasets/pagantibet/normalisation-S2S-training.de-wiktionary-sqlite-normalized
German Wiktionary - Normalized SQLite Database
A fully normalized, production-ready SQLite database of German Wiktionary with complete linguistic information and optimized query performance.
🎯 Key Features
✅ Zero data loss: All information from original Wiktionary preserved
⚡ Lightning-fast queries: Comprehensive indexing (< 5ms typical queries)
🔍 Full grammatical analysis: Complete inflection paradigms, word forms, 185 unique grammatical tags
🔗 Semantic relations:… See the full description on the dataset page: https://huggingface.co/datasets/cstr/de-wiktionary-sqlite-normalized.vietnam-normalize-24kDistillDetect-normalized-traces
DistillDetect — format-normalized teacher traces
Teacher responses from Reference-Based Distillation Detection in LLMs
(arXiv:2607.09692), rewritten so that every
teacher uses the same output format. 7,918 rows across 8 teacher/prompt-set
pairs.
Why this exists
In the released data each teacher emits a structurally different response, so a
student trained on it — and any detector trained to attribute it — can key on
surface format instead of the teacher's actual… See the full description on the dataset page: https://huggingface.co/datasets/francescortu/DistillDetect-normalized-traces.story_generation_reward_train_normal
Reward Training — Normal (EpisodeBench)
This dataset is one of four distribution-controlled reward-training resources released as part of EpisodeBench, a full-cycle benchmarking pipeline for long-form interactive story generation with controllable RL.
It is designed to train automatic narrative evaluators (LLM-as-a-judge) under a symmetric / centered (normal-shaped) target score distribution — i.e., score frequencies are concentrated around the rubric mid-point and decay smoothly… See the full description on the dataset page: https://huggingface.co/datasets/HeAAAAA/story_generation_reward_train_normal.turkish-chat-normalization-mini
Turkish Chat Normalization Mini
turkish-chat-normalization-mini is a web-derived and rule-degraded Turkish text normalization dataset designed for rewriting noisy, informal, unpunctuated, or diacritics-missing Turkish text into cleaner and more readable Turkish.
The dataset does not contain private user messages, chat logs, social media comments, complaint records, or scraped personal conversations. Source sentences are collected from open Turkish web resources, while the input… See the full description on the dataset page: https://huggingface.co/datasets/yagmurtuncer/turkish-chat-normalization-mini.turkish-text-normalization
🇹🇷 Turkish Text Normalization (TN / ITN)
A deterministic, rule-based dataset of Turkish written ↔ spoken pairs for
Text Normalization (TN) and Inverse Text Normalization (ITN) — mapping digit/symbol
forms (1.500 TL, %25, 15.07.2026) to their fully spoken Turkish words
(bin beş yüz lira, yüzde yirmi beş, on beş temmuz iki bin yirmi altı) and back.
This is a common, high-value preprocessing step for Turkish ASR post-processing and
TTS front-ends, where numbers, dates, currencies… See the full description on the dataset page: https://huggingface.co/datasets/yagmurtuncer/turkish-text-normalization.normalized-wikidata
Normalized Wikidata
A preprocessed text-form view of Wikidata, optimised for training language
models or knowledge-graph world models. The goal is a corpus where the
semantic content of Wikidata triples comes through cleanly, with the
catalog-and-identifier clutter that dominates raw Wikidata by volume stripped
out.
License inherits from Wikidata: CC-BY-SA 4.0.
This dataset is the input to a corresponding series of Loka world-model
checkpoints at EmmaLeonhart/loka.
Each snapshot… See the full description on the dataset page: https://huggingface.co/datasets/EmmaLeonhart/normalized-wikidata.turkish-text-normalization-1m
Turkish Text Normalization 1M v2
Kontrollü altı gürültü türüyle Türkçe metin normalizasyon çiftleri.
Doğrulanmış boyut
Train: 980,000
Validation: 10,000
Test: 10,000
Toplam: 1,000,000
Ana görev sütunları: id, noisy_text, normalized_text, noise_type
Provenance
Veri insan mesajlarından, belgelerinden veya web kazımasından alınmamıştır. Tamamı
depodaki üretici koduyla deterministik olarak oluşturulur. Her satırda source_type,
provenance… See the full description on the dataset page: https://huggingface.co/datasets/GoktugD/turkish-text-normalization-1m.kyrgyz-text-normalization
Kyrgyz Text Normalization Dataset
A dataset for training and evaluating Kyrgyz text normalization systems. Released subset accompanying "Kyrgyz Text Normalization: A Comparative Study of Neural and Rule-Based Approaches" (MeLLM Workshop @ ACL 2026).
What is in this release
This is a representative 20,000-pair subset of a larger 1.67M-pair training corpus, plus the full 1,000-example human-verified test set used in the paper.
Split
Examples
Source
Verification… See the full description on the dataset page: https://huggingface.co/datasets/Zarinaaa/kyrgyz-text-normalization.Vietnamese-normThis dataset use Vinorm and Llama to normalize Vietnamese text
For example:
33/4 -> ba mươi ba tháng tư
43 tỷ USD -> bốn mươi ba tỉ đô la
Covid-19 -> covid mười chín
lần thứ VI -> lần thứ sáu
33% -> ba mươi ba phần trăm
U23 -> u hai mươi ba
iPhone 14 -> iphone mười bốn
năm 2023 -> năm hai không hai mươi ba
Amharic_news_Normalized
Dataset Name
Amharic news dataset
Dataset Details
It is a non-normalized version of news dataset crawled from Amharic news websites and from researchers provided in their works.
Dataset Description
The dataset is collected from different news websites and from different researchers crawled Amharic news dataset from different NLP downstream tasks.
News sites like FanaBC, EthiopianReporter, Zehabesha,Esat Amharic, BBC Amharic
are the sources for these news data.… See the full description on the dataset page: https://huggingface.co/datasets/userdavek/Amharic_news_Normalized.Tibetan-normalisation-testdata
Tibetan Normalisation - Test Data
A collection of evaluation datasets for Classical Tibetan text normalisation, containing three distinct test sets designed to assess normalisation systems under different conditions: a manually curated gold-standard set of diplomatic manuscript text, and two synthetic sets of Standard Classical Tibetan text with OCR-based noise applied. Together these test sets allow evaluation across a spectrum from clean, realistic manuscript normalisation to more… See the full description on the dataset page: https://huggingface.co/datasets/pagantibet/Tibetan-normalisation-testdata.TinyPersianStories_normalizedtravelplanner-benchmark-normalized
TravelPlanner Benchmark (Normalized)
Normalized, typed, parquet-first packaging of the TravelPlanner benchmark for planning-centric agent evaluation.
Upstream dataset: osunlp/TravelPlanner
Upstream code: OSU-NLP-Group/TravelPlanner
Paper: TravelPlanner: A Benchmark for Real-World Planning with Language Agents
1) What is included
This dataset repo contains:
benchmark config (train/validation/test) in typed parquet.
reference_entries config: flattened reference-info… See the full description on the dataset page: https://huggingface.co/datasets/tuandunghcmut/travelplanner-benchmark-normalized.turkish-datetime-normalization-500k
Turkish Datetime Normalization 500K v2
Türkçe tarih-saat ifadelerini ISO-8601 ve Europe/Istanbul saat dilimine eşler.
Doğrulanmış boyut
Train: 490,000
Validation: 5,000
Test: 5,000
Toplam: 500,000
Ana görev sütunları: id, text, normalized_datetime, timezone
Provenance
Veri insan mesajlarından, belgelerinden veya web kazımasından alınmamıştır. Tamamı
depodaki üretici koduyla deterministik olarak oluşturulur. Her satırda source_type,
provenance… See the full description on the dataset page: https://huggingface.co/datasets/GoktugD/turkish-datetime-normalization-500k.dair-ai-emotion-normalized-instruction-input-output
dair-ai emotion | normalized
Summary
Dataset ID: 143
Type: normalized
Rows: 16,000
Source: dair-ai/emotion
Dataset Sources
#143 dair-ai emotion | normalized [normalized | 16,000 rows]
Notes
Edited and Exported from the Kitsune Training Suite (Forge)
Review the dataset artifact and metadata before publishing.
Citation > via dair-ai
@inproceedings{saravia-etal-2018-carer,
title = "{CARER}: Contextualized Affect… See the full description on the dataset page: https://huggingface.co/datasets/atrevidasadia/dair-ai-emotion-normalized-instruction-input-output.dostoevsky_frontier_3k_normalized
dostoevsky_frontier_3k_normalized
Нормализованная версия 0x7o/dostoevsky_frontier_3k.
Нормализация
Устранены пунктуационные shortcut-ы, позволяющие модели различать chosen/rejected
по артефактам форматирования вместо стиля.
Общие (chosen + rejected)
ё → е по словарю (книги не используют ё, AI всегда использует — 87.8% accuracy)
\xa0 (неразрывный пробел) → обычный пробел
… (U+2026) → ... (три точки)
– (en dash) → — (em dash)
!.. → !..., ?.. → ?...… See the full description on the dataset page: https://huggingface.co/datasets/0x7o/dostoevsky_frontier_3k_normalized.nuclei-template-generation-dataset-2.3K
nuclei-template-generation-dataset-2.3K
Description:
A specialized instruction-tuning dataset of 2350 examples for training large language models to generate Nuclei YAML templates. Each example consists of a fixed instruction, a structured JSON input describing a vulnerability (CVE, product, HTTP details, detection logic), and the corresponding valid Nuclei template as output. The dataset was constructed from the official Nuclei Templates repository (HTTP… See the full description on the dataset page: https://huggingface.co/datasets/NormanRey/nuclei-template-generation-dataset-2.3K.