datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
flan2021-full
Task Name
FLAN-2021 -> 70
{
"ag_news_subset": 108497,
"ai2_arc/ARC-Challenge": 829,
"ai2_arc/ARC-Easy": 1927,
"aeslc": 13187,
"anli/r1": 15361,
"anli/r2": 41133,
"anli/r3": 91048,
"bool_q": 8343,
"cnn_dailymail": 259607,
"coqa": 6456,
"cosmos_qa": 22996,
"definite_pronoun_resolution": 1079,
"drop": 70045,
"fix_punct": 25690,
"gem/common_gen": 60936,
"gem/dart": 56724,
"gem/e2e_nlg": 30337,
"gem/web_nlg_en": 31899… See the full description on the dataset page: https://huggingface.co/datasets/aslawliet/flan2021-full.Icelandic-Flan
Icelandic FLAN
Icelandic instruction-following data, built by pairing licensed, human-written Icelandic
texts with deterministic instruction templates.
Status
16 sources · 46 tasks · 602,057 rows · 45.6M response characters.
Source
Register
Licence
Rows
Response chars
Share
umbodsmadur
administrative law — Ombudsman
art-9
3,914
9,265,216
20.3%
igc_news
journalism
CC BY 4.0
27,711
8,984,257
19.7%
rafbokavefur
literary — diacritic restoration over… See the full description on the dataset page: https://huggingface.co/datasets/Frejams/Icelandic-Flan.faroese-flan
Faroese FLAN
Faroese instruction-following data, built by pairing licensed or public-domain, human-written Faroese
texts with deterministic instruction templates. The sibling of the Icelandic collection, with the same
row schema and the same release checks.
Status
13 sources · 31 tasks · 1,000,543 rows · 34.0M response characters.
Source
Register
Licence
Rows
Response chars
Share
logir
consolidated law
public-domain-fo-p9
29,178
14,798,010
43.6%… See the full description on the dataset page: https://huggingface.co/datasets/Frejams/faroese-flan.flan_2022_350k
Dataset Card for Flan-2022 Subsample (Flan 350K)
Dataset Summary
This dataset was used in the paper on the origins of cognitive biases in LLMs:
"Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs"https://arxiv.org/abs/2507.07186
This is a 350,000-example subsample of the original FLAN 2022 instruction-tuning dataset (https://arxiv.org/abs/2210.11416). It was created to provide a balanced, computationally efficient variant… See the full description on the dataset page: https://huggingface.co/datasets/itay1itzhak/flan_2022_350k.flan-routing-MoE-datasetflan-v2-converted-enflan_v2_converted filtered down to English only using GlotLID, derived and inheriting the license from OpenOrca FLAN.
flanv2_cot_dedepulicated
FLAN v2 Cot Deduplicated Dataset
Data Preprocessing
Remove instructions with less than 100 tokens in 'targets'.
Dedepulicate Dataset using cosine similarity with a threshold of 0.95.
Code
Github repo : https://github.com/AJlearner46/Deduplicate-flanv2-finetune-LLaMa3-
Acknowledgments
The original dataset is provided by SirNeural/flan_v2.
Tokenizer used: bert-base-uncased from Hugging Face.