polyglot
Datasets
All datasets matching “polyglot”Fact-Completion
Dataset Card
Homepage: https://bit.ly/ischool-berkeley-capstone
Repository: https://github.com/daniel-furman/Capstone
Point of Contact: daniel_furman@berkeley.edu
Dataset Summary
This is the dataset for Polyglot or Not?: Measuring Multilingual Encyclopedic Knowledge Retrieval from Foundation Language Models.
Test Description
Given a factual association such as The capital of France is Paris, we determine whether a model adequately "knows" this… See the full description on the dataset page: https://huggingface.co/datasets/Polyglot-or-Not/Fact-Completion.PolygloToxicityPrompts
PolygloToxicityPrompts
Dataset Summary
A multilingual toxicity evaluation benchmark curated from web text.
We prepared 3 splits: ptp-full, ptp-small, and wildchat containining 25K, 5K and 1K prompts per language respectively.
The wildchat split is created using AI2's WildChat dataset.
How do I download this?
Using 🤗 Datasets
from datasets import load_dataset
# English only
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/ToxicityPrompts/PolygloToxicityPrompts.Polyglot-Thoughts-SFT-Collection
Polyglot Thoughts SFT Collection
Polyglot Thoughts SFT Collection is a large-scale supervised fine-tuning (SFT)
corpus for reasoning-oriented language models. It combines, filters, deduplicates,
and language-extends a broad set of public reasoning datasets into a single uniform
schema centred on chain-of-thought reasoning traces.
The final corpus contains 23,896,757 examples and roughly 123 billion tokens,
spanning six languages (English, German, French, Italian, Spanish… See the full description on the dataset page: https://huggingface.co/datasets/ahmad21omar/Polyglot-Thoughts-SFT-Collection.polyglot-ko-tokenizer-corpus
Dataset Card for "polyglot-ko-tokenizer-corpus"
More Information needed
aider_polyglotpolyglot_nerPolyglot-NER
A training dataset automatically generated from Wikipedia and Freebase the task
of named entity recognition. The dataset contains the basic Wikipedia based
training data for 40 languages we have (with coreference resolution) for the task of
named entity recognition. The details of the procedure of generating them is outlined in
Section 3 of the paper (https://arxiv.org/abs/1410.3791). Each config contains the data
corresponding to a different language. For example, "es" includes only spanish examples.