datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
c4_t5_corrupted_seqlen256
Dataset Card for "c4_t5_corrupted_seqlen256"
More Information needed
openwebtext-t5Mintaka_Graph_Features_T5-xl-ssm
Dataset Card for "Mintaka_Graph_Features_T5-xl-ssm"
More Information needed
c4-t5-ragged
C4, T5 tokenized, in ragged array format
Processed distribution of Google's C4 dataset: a colossal, cleaned version of Common Crawl's web crawl corpus.
Uses the text data from allenai/c4.
Includes en subset only.
T5 tokenizer was applied to the text.Distributed as a ragged array.
Converted via json_to_ragged.py.
Download size of all shards:
Split
Data+Lengths Size
Divided across n Shards
Typical shard size: data.npy
Typical shard size: len.npy
Train
293G
1024
344M
1.4M… See the full description on the dataset page: https://huggingface.co/datasets/Birchlabs/c4-t5-ragged.xsum_validation_t5capstone_sakuga_iblip_t5_embeddingslaion-pop-vae-t5c4_t5_pretrain
Dataset Card for "c4_t5_pretrain"
More Information needed
c4_t5
Dataset Card for "c4_t5"
More Information needed
flan-t5-large-embed-refinedwebAll of the data together is around 81.3GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through google/flan-t5-base.
Structure:
{
"encoding": List, shaped (512, 1024) aka (tokens, d_model),
"text": String, the original text that was encoded,
"attention_mask": List, binary mask to pass to your model with encoding to not attend to pad tokens
}
t5-small-october-wikipedia-2022-tokenized-512
Dataset Card for "t5-small-october-wikipedia-2022-tokenized-512"
More Information needed
gtr-t5-xxl-wikipedia-psgs_w100-indexwmt14_de-en_validation_t5c4-t5-subset2a_t5_page14_batch1c4_t5_packed
Dataset Card for "c4_t5_packed"
More Information needed
xsum_train_t5a_t5_page10_batch2flan-t5-base-embed-refinedwebAll of the data together is around 61GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through google/flan-t5-base.
Structure:
{
"encoding": List, shaped (512, 768) aka (tokens, d_model),
"text": String, the original text that was encoded,
"attention_mask": List, binary mask to pass to your model with encoding to not attend to pad tokens
}
KGQA_T5-xl-ssm
Dataset Card for "KGQA_T5-xl-ssm"
More Information needed
a_t5_page16_batch1a_t5_page15_batch1t5gemma2-indonesia-instruct-v1
T5Gemma-2 Indonesian Instruct — Mono-Repo
Satu repositori dataset HF untuk seluruh data pelatihan T5-Gemma-2 bahasa Indonesia.
Diorganisasi per fungsi (fondasi → spesifik → preferensi) dengan folder/subfolder,
setiap config = folder dan berisi split train + validation (80:20) di level percakapan.
Struktur (by fungsi)
t5gemma2-indonesia-instruct-v1/
├── README.md
├── manifest.json
├── chat_idx_map.json
├── foundation/ ← FASE 1 · fondasi Bahasa… See the full description on the dataset page: https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-instruct-v1.a_t5_page12_batch1a_t5_page17_batch2c4_t5_10m
Dataset Card for "c4_t5_10m"
More Information needed
a_t5_page9_batch1epidemic_sound_effects_t5_debiasedc4-t5-subsetflan-t5-small-embed-refinedwebAll of the data together is around 41GB. It's the last hidden states of 131,072 samples from refinedweb padded/truncated to 512 tokens on the left, fed through google/flan-t5-small.
Structure:
{
"encoding": List, shaped (512, 512) aka (tokens, d_model),
"text": String, the original text that was encoded,
"attention_mask": List, binary mask to pass to your model with encoding to not attend to pad tokens
}
just a tip, you cannot load this with the RAM in the free ver of google colab, not… See the full description on the dataset page: https://huggingface.co/datasets/crumb/flan-t5-small-embed-refinedweb.