datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
FLAN🍮 The WHOLE FLAN Collection! 🍮
Overview
This repository includes the full dataset from the FLAN Collection, totalling ~300GB as parquets.
Generated using the official seqio templating from the Google FLAN Collection GitHub repo.
The data is subject to all the same licensing of the component datasets.
To keep up with our continued work on OpenOrca and other exciting research, find our Discord here:
https://AlignmentLab.ai
Motivation
This work was done as part of… See the full description on the dataset page: https://huggingface.co/datasets/Open-Orca/FLAN.flanThis is a repreprocessed version of the FLAN dataset with any updates that have been made to the FLAN datasets since the release of the original FLAN. The script is available here.
Tasks:
{'aeslc_10templates',
'ag_news_subset_10templates',
'anli_r1_10templates',
'anli_r2_10templates',
'anli_r3_10templates',
'arc_challenge_10templates',
'arc_easy_10templates',
'bool_q_10templates',
'cb_10templates',
'cnn_dailymail_10templates',
'cola_10templates',
'common_gen_10templates'… See the full description on the dataset page: https://huggingface.co/datasets/Muennighoff/flan.flan-v2
Dataset Card for "flan-v2"
More Information needed
sinhala-flanflan_v2
Dataset Card for Flan V2
Dataset Summary
This is a processed version of the Flan V2 dataset.
I'm not affiliated with the creators, I'm just releasing the files in an easier-to-access format after processing.
The authors of the Flan Collection recommend experimenting with different mixing ratio's of tasks to get optimal results downstream.
Setup Instructions
Here are the steps I followed to get everything working:
Build AESLC and WinoGrande datasets… See the full description on the dataset page: https://huggingface.co/datasets/SirNeural/flan_v2.flan2022
Dataset Card for "flan2022"
More Information needed
flanv2FLANv2-without-T0flan-t5-large-embed-refinedwebAll of the data together is around 81.3GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through google/flan-t5-base.
Structure:
{
"encoding": List, shaped (512, 1024) aka (tokens, d_model),
"text": String, the original text that was encoded,
"attention_mask": List, binary mask to pass to your model with encoding to not attend to pad tokens
}
flan2021_coreference_raw
Flan 2021 Coreference Tasks
Project: https://github.com/google-research/FLAN/tree/main/flan/v2
Data source: DataProvenanceInitiative/flan2021_submix_original
Details
This dataset contains all coreference examples that were included in the Flan 2022 collection which were orignally included in Flan 2021.
The data is copied from the preprocessed Flan2021 dataset at DataProvenanceInitiative/flan2021_submix_original.
COREFERENCE_TASK_NAMES = {… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/flan2021_coreference_raw.flan-10k-flat
Dataset Card for "flan-10k-flat"
More Information needed
flan-v2-hf
flan-v2: hf datasets format
https://hf.co/datasets/philschmid/flanv2 directly loaded into hf datasets parquet files for easier streaming, etc.
[!NOTE]
There are two other configs besides the default, en-targets and targets-3w
en-targets: the targets column filtered with fasttext-langdetect for en and score >= 0.65
targets-3w: the targets column filtered for 3 or more words
flan-cotPhatgoose_flanv2_offlineflan-swedishflanv2
Fork of SirNeural/flan_v2
just in case it gets deleted.
Dataset Card for Flan V2
Dataset Summary
This is a processed version of the Flan V2 dataset.
I'm not affiliated with the creators, I'm just releasing the files in an easier-to-access format after processing.
The authors of the Flan Collection recommend experimenting with different mixing ratio's of tasks to get optimal results downstream.
This current version I've processed is missing a few… See the full description on the dataset page: https://huggingface.co/datasets/philschmid/flanv2.FLAN-compressed-plusplus
flan-compressed ++
FLAN-compressed with additional tasks added, mostly programming-related. As per Google's flan-v2 README, they seemingly excluded most (all?) programming/code tasks from the data they published.
README WIP
flan-dolminoflan2021-full
Task Name
FLAN-2021 -> 70
{
"ag_news_subset": 108497,
"ai2_arc/ARC-Challenge": 829,
"ai2_arc/ARC-Easy": 1927,
"aeslc": 13187,
"anli/r1": 15361,
"anli/r2": 41133,
"anli/r3": 91048,
"bool_q": 8343,
"cnn_dailymail": 259607,
"coqa": 6456,
"cosmos_qa": 22996,
"definite_pronoun_resolution": 1079,
"drop": 70045,
"fix_punct": 25690,
"gem/common_gen": 60936,
"gem/dart": 56724,
"gem/e2e_nlg": 30337,
"gem/web_nlg_en": 31899… See the full description on the dataset page: https://huggingface.co/datasets/aslawliet/flan2021-full.flan-t5-base-embed-refinedwebAll of the data together is around 61GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through google/flan-t5-base.
Structure:
{
"encoding": List, shaped (512, 768) aka (tokens, d_model),
"text": String, the original text that was encoded,
"attention_mask": List, binary mask to pass to your model with encoding to not attend to pad tokens
}
mini-sinhala-flanflan-ul2-tinystoriesAround a quarter of a million examples generated from Flan-UL2 (20b) with the prompt "Write a short story using the vocabulary of a first-grader." to be used in an experimental curriculum learning setting. I had to checkpoint every 1024 examples to mitigate the program slowing down due to memory usage. This was run in bf16 on an RTXA6000 with the following settings:
top_k = random between (40, 128)
temperature = random between (0.6, 0.95)
max_length = 128
batch_size = 32
I wanted a less… See the full description on the dataset page: https://huggingface.co/datasets/crumb/flan-ul2-tinystories.flan_v2_convertedThis is a converted version of the Flan dataset into Tulu SFT training format.
The conversion script can be found in our open-instruct repo.
The conversion took the following parameters:
apply_keyword_filters: True
apply_empty_message_filters: True
push_to_hub: True
hf_entity: ai2-adapt-dev
converted_dataset_name: flan_v2_converted
local_save_dir: ./data/sft/flan
The original FLAN dataset needs extensive efforts to be regenerated, so we are using a reproduced version by the OpenOrca… See the full description on the dataset page: https://huggingface.co/datasets/ai2-adapt-dev/flan_v2_converted.flanv2_subsampleflan-ul2-tinystories-complexAround a quarter of a million examples generated from Flan-UL2 (20b) with the prompt "Write a complex short story using the vocabulary of a third-grader." to be used in an experimental curriculum learning setting. I had to checkpoint every 1024 examples to mitigate the program slowing down due to memory usage. This was run in bf16 on an RTXA6000 with the following settings:
top_k = random between (40, 128)
temperature = random between (0.6, 0.95)
max_length = 128
batch_size = 32
I wanted a… See the full description on the dataset page: https://huggingface.co/datasets/crumb/flan-ul2-tinystories-complex.dolmino-mix-1124-flanflan-subsets-deduped
flan subsets: deduped
[!IMPORTANT]
see config all for the aggregated & deduped dataset
all configs/subsets have columns inputs and targets
deduped on inputs
filtered for lang en if contents more than 5 chars.
filter out any row with less than 1 char for either column
clean-text applied to both columns
dedup command
deduped with:
python -m text_dedup.minhash \
--path $ds_name \
--name $dataset_config \
--split $data_split \
--cache_dir "./cache" \… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/flan-subsets-deduped.Icelandic-Flan
Icelandic FLAN
Icelandic instruction-following data, built by pairing licensed, human-written Icelandic
texts with deterministic instruction templates.
Status
16 sources · 46 tasks · 602,057 rows · 45.6M response characters.
Source
Register
Licence
Rows
Response chars
Share
umbodsmadur
administrative law — Ombudsman
art-9
3,914
9,265,216
20.3%
igc_news
journalism
CC BY 4.0
27,711
8,984,257
19.7%
rafbokavefur
literary — diacritic restoration over… See the full description on the dataset page: https://huggingface.co/datasets/Frejams/Icelandic-Flan.flan-t5-small-embed-refinedwebAll of the data together is around 41GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through google/flan-t5-small.
Structure:
{
"encoding": List, shaped (512, 512) aka (tokens, d_model),
"text": String, the original text that was encoded,
"attention_mask": List, binary mask to pass to your model with encoding to not attend to pad tokens
}
just a tip, you cannot load this with the RAM in the free ver of google colab, not… See the full description on the dataset page: https://huggingface.co/datasets/crumb/flan-t5-small-embed-refinedweb.vision-flan_191-task_1k
Dataset Card for "vision-flan_191-task_1k"
More Information needed