datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Fact-Completion
Dataset Card
Homepage: https://bit.ly/ischool-berkeley-capstone
Repository: https://github.com/daniel-furman/Capstone
Point of Contact: daniel_furman@berkeley.edu
Dataset Summary
This is the dataset for Polyglot or Not?: Measuring Multilingual Encyclopedic Knowledge Retrieval from Foundation Language Models.
Test Description
Given a factual association such as The capital of France is Paris, we determine whether a model adequately "knows" this… See the full description on the dataset page: https://huggingface.co/datasets/Polyglot-or-Not/Fact-Completion.PolygloToxicityPrompts
PolygloToxicityPrompts
Dataset Summary
A multilingual toxicity evaluation benchmark curated from web text.
We prepared 3 splits: ptp-full, ptp-small, and wildchat containining 25K, 5K and 1K prompts per language respectively.
The wildchat split is created using AI2's WildChat dataset.
How do I download this?
Using 🤗 Datasets
from datasets import load_dataset
# English only
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/ToxicityPrompts/PolygloToxicityPrompts.Polyglot-Thoughts-SFT-Collection
Polyglot Thoughts SFT Collection
Polyglot Thoughts SFT Collection is a large-scale supervised fine-tuning (SFT)
corpus for reasoning-oriented language models. It combines, filters, deduplicates,
and language-extends a broad set of public reasoning datasets into a single uniform
schema centred on chain-of-thought reasoning traces.
The final corpus contains 23,896,757 examples and roughly 123 billion tokens,
spanning six languages (English, German, French, Italian, Spanish… See the full description on the dataset page: https://huggingface.co/datasets/ahmad21omar/Polyglot-Thoughts-SFT-Collection.wikipedia-language-snippets-filtered
Wikipedia Snippets (Filtered)
Filtered sentence snippets in Wikipedia, by taking the first 60% of an article after filtering for stubs. Minor Latin groups are additionally filtered again for English leakage.
Sentences are mostly filtered out for non matching scripts, such as Arabic in a Cyrllic language.
Files
Each file is in this format for languages in ISO 639 2-letter codes:
train/en/en.parquet
train/es/es.parquet
From wikimedia/wikipedia
Licensing… See the full description on the dataset page: https://huggingface.co/datasets/polyglot-tagger/wikipedia-language-snippets-filtered.Polyglot-Thoughts-RL-Collection
Polyglot Thoughts RL Collection
Polyglot Thoughts RL Collection is a large-scale, curated corpus for
reinforcement learning from verifiable rewards (RLVR) of reasoning-oriented
language models. It combines, filters, normalises, and deduplicates a broad set
of public RL datasets into a single uniform schema in which every row carries a
machine-verifiable ground-truth signal — math equivalence, code execution,
Prolog rule induction, schema validation, multiple-choice… See the full description on the dataset page: https://huggingface.co/datasets/ahmad21omar/Polyglot-Thoughts-RL-Collection.PolyglotTeachers-SFT-Synth-Data
Website: ljvmiranda921.github.io/polyglot-teachers/
PolyglotTeachers-SFT-Synth
This dataset contains synthetic supervised fine-tuning examples generated by the best teacher we found in the paper Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation, where we systematically characterize what makes a good teacher model.
It contains examples across six languages: Arabic, Czech, German, Indonesian, Japanese, Spanish, and Tagalog. Note: In… See the full description on the dataset page: https://huggingface.co/datasets/ljvmiranda921/PolyglotTeachers-SFT-Synth-Data.PolygloToxicityPrompts_permissive
Polyglo PTP Permissive
Permissive-source subset of the PTP configs from ToxicityPrompts/PolygloToxicityPrompts, filtered using the local URL/source permissiveness rules in this workspace. WildChat configs are not included because they do not provide row-level source URLs.
Source split: full for ptp-* configs.
Rows scanned: 425,000. Rows kept: 1,649.
See polyglo_ptp_permissive_stats_620972.json for kept/discard reason distributions and per-language counts.
polyglot-996