datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Fact-Completion
Dataset Card
Homepage: https://bit.ly/ischool-berkeley-capstone
Repository: https://github.com/daniel-furman/Capstone
Point of Contact: daniel_furman@berkeley.edu
Dataset Summary
This is the dataset for Polyglot or Not?: Measuring Multilingual Encyclopedic Knowledge Retrieval from Foundation Language Models.
Test Description
Given a factual association such as The capital of France is Paris, we determine whether a model adequately "knows" this… See the full description on the dataset page: https://huggingface.co/datasets/Polyglot-or-Not/Fact-Completion.PolygloToxicityPrompts
PolygloToxicityPrompts
Dataset Summary
A multilingual toxicity evaluation benchmark curated from web text.
We prepared 3 splits: ptp-full, ptp-small, and wildchat containining 25K, 5K and 1K prompts per language respectively.
The wildchat split is created using AI2's WildChat dataset.
How do I download this?
Using 🤗 Datasets
from datasets import load_dataset
# English only
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/ToxicityPrompts/PolygloToxicityPrompts.Polyglot-Thoughts-SFT-Collection
Polyglot Thoughts SFT Collection
Polyglot Thoughts SFT Collection is a large-scale supervised fine-tuning (SFT)
corpus for reasoning-oriented language models. It combines, filters, deduplicates,
and language-extends a broad set of public reasoning datasets into a single uniform
schema centred on chain-of-thought reasoning traces.
The final corpus contains 23,896,757 examples and roughly 123 billion tokens,
spanning six languages (English, German, French, Italian, Spanish… See the full description on the dataset page: https://huggingface.co/datasets/ahmad21omar/Polyglot-Thoughts-SFT-Collection.polyglot-ko-tokenizer-corpus
Dataset Card for "polyglot-ko-tokenizer-corpus"
More Information needed
aider_polyglotpolyglot_nerPolyglot-NER
A training dataset automatically generated from Wikipedia and Freebase the task
of named entity recognition. The dataset contains the basic Wikipedia based
training data for 40 languages we have (with coreference resolution) for the task of
named entity recognition. The details of the procedure of generating them is outlined in
Section 3 of the paper (https://arxiv.org/abs/1410.3791). Each config contains the data
corresponding to a different language. For example, "es" includes only spanish examples.polyglotoxicitypromptswikipedia-language-snippets-filtered
Wikipedia Snippets (Filtered)
Filtered sentence snippets in Wikipedia, by taking the first 60% of an article after filtering for stubs. Minor Latin groups are additionally filtered again for English leakage.
Sentences are mostly filtered out for non matching scripts, such as Arabic in a Cyrllic language.
Files
Each file is in this format for languages in ISO 639 2-letter codes:
train/en/en.parquet
train/es/es.parquet
From wikimedia/wikipedia
Licensing… See the full description on the dataset page: https://huggingface.co/datasets/polyglot-tagger/wikipedia-language-snippets-filtered.finetranslations-filtered
Files
Each file is in this format for languages in ISO 639 2-letter codes:
train/en/en.parquet
train/es/es.parquet
From HuggingFaceFW/finetranslations
Licensing Information
The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.
Citation Information
@misc{penedo2026finetranslations,
title={FineTranslations},
author={Guilherme… See the full description on the dataset page: https://huggingface.co/datasets/polyglot-tagger/finetranslations-filtered.Polyglot-Thoughts-RL-Collection
Polyglot Thoughts RL Collection
Polyglot Thoughts RL Collection is a large-scale, curated corpus for
reinforcement learning from verifiable rewards (RLVR) of reasoning-oriented
language models. It combines, filters, normalises, and deduplicates a broad set
of public RL datasets into a single uniform schema in which every row carries a
machine-verifiable ground-truth signal — math equivalence, code execution,
Prolog rule induction, schema validation, multiple-choice… See the full description on the dataset page: https://huggingface.co/datasets/ahmad21omar/Polyglot-Thoughts-RL-Collection.PolyglotTeachers-SFT-Synth-Data
Website: ljvmiranda921.github.io/polyglot-teachers/
PolyglotTeachers-SFT-Synth
This dataset contains synthetic supervised fine-tuning examples generated by the best teacher we found in the paper Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation, where we systematically characterize what makes a good teacher model.
It contains examples across six languages: Arabic, Czech, German, Indonesian, Japanese, Spanish, and Tagalog. Note: In… See the full description on the dataset page: https://huggingface.co/datasets/ljvmiranda921/PolyglotTeachers-SFT-Synth-Data.tatoeba-filtered
Files
Each file is in this format for languages in ISO 639 2-letter codes:
train/en/en.parquet
train/es/es.parquet
aider_polyglot_a3_rl_DCAgent_inferredbugs_sandboxes_verifier_55_8B_20260526_230125aider_polyglot_GLM_4_7_swesmith_sandboxes_with_tests_oracle_verified_120s_maxepf48469bfaider_polyglot_a3_rl_DCAgent_selfinstruct_naive_sandboxes_2_verified_70_8B_202690541bcechad-polyglot-runs
chad run data
Run output from the benchmarks of chad, a local
coding agent for Apple Silicon, kept out of the code repository. Every polyglot run here is
a row of benchmarks/polyglot/RUNS.md
with the sha256 of its trials.jsonl, and benchmarks/polyglot/fetch.py --label <label>
downloads one and refuses it if the hash differs.
path
what
polyglot/<label>/
one polyglot run: meta.json (chad version and commit, model, context limit, every CHAD_* variable), trials.jsonl (one… See the full description on the dataset page: https://huggingface.co/datasets/nathansutton/chad-polyglot-runs.MADLAD_CulturaX_cleanedPolyglotAudio
Citation
If you use this dataset in your research or downstream work, please cite:
@misc{polyglot_audio_2026,
author = {Fernandes, Reuben Chagas},
title = {PolyglotAudio: Multilingual Audio Pre-training Corpus},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/Reubencf/PolyglotAudio}}
}
APA-style:
Reuben Chagas Fernandes (2026). PolyglotAudio: Multilingual Audio Pre-training Corpus [Dataset]. Hugging… See the full description on the dataset page: https://huggingface.co/datasets/Reubencf/PolyglotAudio.DCAgent2_aider_polyglot_DCAgent2_swesmith-stack-reason_20260127_005703nlp-noise-snippets
Synthetic Noise Pool
For Text Classification purposes, as many models may consider code snippets, html artifacts, and math as "English".
Around 50K are latex snippets from im2latex-100k
details_EleutherAI__polyglot-ko-12.8b
Dataset Card for Evaluation run of EleutherAI/polyglot-ko-12.8b
Dataset Summary
Dataset automatically created during the evaluation run of model EleutherAI/polyglot-ko-12.8b on the Open LLM Leaderboard.
The dataset is composed of 64 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train"… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_EleutherAI__polyglot-ko-12.8b.DCAgent2_aider_polyglot_DCAgent2_nl2bash-swesmith-reason_20260127_005703details_beomi__KoAlpaca-Polyglot-5.8B
Dataset Card for Evaluation run of beomi/KoAlpaca-Polyglot-5.8B
Dataset Summary
Dataset automatically created during the evaluation run of model beomi/KoAlpaca-Polyglot-5.8B on the Open LLM Leaderboard.
The dataset is composed of 64 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 3 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train"… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_beomi__KoAlpaca-Polyglot-5.8B.polyglotfake-realdetails_macadeliccc__laser-polyglot-4x7b
Dataset Card for Evaluation run of macadeliccc/laser-polyglot-4x7b
Dataset automatically created during the evaluation run of model macadeliccc/laser-polyglot-4x7b on the Open LLM Leaderboard.
The dataset is composed of 63 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_macadeliccc__laser-polyglot-4x7b.polyglot_paylods_datasets
Polyglot Payloads Dataset for Cybersecurity Training
Overview
This dataset, polyglot_payloads.jsonl, is a curated collection of 500 polyglot payloads designed for training AI models in cybersecurity, specifically for red team operations and vulnerability detection. The dataset includes payloads targeting common web vulnerabilities such as Cross-Site Scripting (XSS), SQL Injection (SQLi), Local File Inclusion (LFI), Remote Code Execution (RCE), and Server-Side Template… See the full description on the dataset page: https://huggingface.co/datasets/darkknight25/polyglot_paylods_datasets.DCAgent2_aider_polyglot_DCAgent_freelancer-long-instruction-filter_Qwen3-8B_2021a649aaeDCAgent2_aider_polyglot_DCAgent_tbench_oracle_solutions_terminus_20260125_202032details_macadeliccc__polyglot-math-4x7b
Dataset Card for Evaluation run of macadeliccc/polyglot-math-4x7b
Dataset automatically created during the evaluation run of model macadeliccc/polyglot-math-4x7b on the Open LLM Leaderboard.
The dataset is composed of 63 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_macadeliccc__polyglot-math-4x7b.PolygloToxicityPrompts_permissive
Polyglo PTP Permissive
Permissive-source subset of the PTP configs from ToxicityPrompts/PolygloToxicityPrompts, filtered using the local URL/source permissiveness rules in this workspace. WildChat configs are not included because they do not provide row-level source URLs.
Source split: full for ptp-* configs.
Rows scanned: 425,000. Rows kept: 1,649.
See polyglo_ptp_permissive_stats_620972.json for kept/discard reason distributions and per-language counts.