datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
the_pile_deduplicatedthe-pile-splitted
Dataset description
The pile is an 800GB dataset of english text
designed by EleutherAI to train large-scale language models. The original version of
the dataset can be found here.
The dataset is divided into 22 smaller high-quality datasets. For more information
each of them, please refer to the datasheet for the pile.
However, the current version of the dataset, available on the Hub, is not splitted accordingly.
We had to solve this problem in order to improve the user… See the full description on the dataset page: https://huggingface.co/datasets/ArmelR/the-pile-splitted.the_pile_github
Dataset Card for The Pile GitHub
Dataset Summary
This is the GitHub subset of EleutherAi/The Pile dataset and contains GitHub repositories. The programming languages are identified using the guesslang library. A total of 54 programming languages are included in the dataset.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
The following languages are covered by the dataset:
'Assembly', 'Batchfile', 'C', 'C#', 'C++', 'CMake'… See the full description on the dataset page: https://huggingface.co/datasets/andstor/the_pile_github.the_pile_openwebtext2
Dataset Card for "the_pile_openwebtext2"
More Information needed
the_pile_books3_minus_gutenberg
Dataset Card for "the_pile_books3_minus_gutenberg"
More Information needed
EleutherAI_the_pile_deduplicatedSince The Pile was removed from the original site, I'm worried this dataset might be taken down too. Putting it here just in case.
Original repo: https://huggingface.co/datasets/EleutherAI/the_pile_deduplicated
scaling_mia_the_pile_00_OpenWebText2scaling_mia_the_pile_00_Pile-CCthe-pile-pubmed-abstracts-refined-by-data-juicer
The Pile -- PubMed Abstracts (refined by Data-Juicer)
A refined version of PubMed Abstracts dataset in The Pile by Data-Juicer. Removing some "bad" samples from the original dataset to make it higher-quality.
This dataset is usually used to pretrain a Large Language Model.
Notice: Here is a small subset for previewing. The whole dataset is available here (About 24G).
Dataset Information
Number of samples: 371,331 (Keep ~99.55% from the original dataset)… See the full description on the dataset page: https://huggingface.co/datasets/datajuicer/the-pile-pubmed-abstracts-refined-by-data-juicer.scaling_mia_the_pile_00_PubMed_Centralthe_pile_arxiv_50k_sampleCreated with this script https://gist.github.com/HaritzPuerto/3543a70b5bfe50c8fec6977c716efe67
the_pile_arxiv_1k_sampleSample of the arxiv partition of The Pile.
The training set is just a random sample of 1000 documents from the 00.jsonl.zst (the first file in The Pile; it seems each jsonl.zst file is already a random sample).
The validation and test set are the full sets.
Statistics
Training Set
Mean number of tokens: 14588.022
Std number of tokens: 26015.51379449416
Max number of tokens: 746616
Min number of tokens: 33
scaling_mia_the_pile_00_arxivThis dataset includes all arxiv documents from the 00.jsonl.zst partition of The Pile. It was created with this script:
pile_path = "data/the_pile/train/00.jsonl.zst"
with zstd.open(pile_path, 'r') as fr:
with open("/tmp/arxiv.jsonl", "w") as fw:
for i, line in enumerate(tqdm(fr)):
doc = json.loads(line)
source = doc['meta']['pile_set_name']
if source == "ArXiv":
fw.write(json.dumps(doc) + "\n")
The validation and test sets are… See the full description on the dataset page: https://huggingface.co/datasets/parameterlab/scaling_mia_the_pile_00_arxiv.small-the_pile
Dataset Card for "small-the_pile"
More Information needed
scaling_mia_the_pile_00_wikiThis dataset includes all Wikipedia documents from the 00.jsonl.zst partition of The Pile. It was created with this script:
pile_path = "data/the_pile/train/00.jsonl.zst"
with zstd.open(pile_path, 'r') as fr:
with open("/tmp/wiki.jsonl", "w") as fw:
for i, line in enumerate(tqdm(fr)):
doc = json.loads(line)
source = doc['meta']['pile_set_name']
if source == "Wikipedia (en)":
fw.write(json.dumps(doc) + "\n")
The validation and… See the full description on the dataset page: https://huggingface.co/datasets/parameterlab/scaling_mia_the_pile_00_wiki.scaling_mia_the_pile_00_Enron_Emailsthe-pile-pubmed-central-refined-by-data-juicer
The Pile -- PubMed Central (refined by Data-Juicer)
A refined version of PubMed Central dataset in The Pile by Data-Juicer. Removing some "bad" samples from the original dataset to make it higher-quality.
This dataset is usually used to pretrain a Large Language Model.
Notice: Here is a small subset for previewing. The whole dataset is available here (About 83G).
Dataset Information
Number of samples: 2,694,860 (Keep ~86.96% from the original dataset)… See the full description on the dataset page: https://huggingface.co/datasets/datajuicer/the-pile-pubmed-central-refined-by-data-juicer.scaling_mia_the_pile_00_Gutenberg_PG-19scaling_mia_the_pile_00_Githubthe-pile-uspto-refined-by-data-juicer
The Pile -- USPTO (refined by Data-Juicer)
A refined version of USPTO dataset in The Pile by Data-Juicer. Removing some "bad" samples from the original dataset to make it higher-quality.
This dataset is usually used to pretrain a Large Language Model.
Notice: Here is a small subset for previewing. The whole dataset is available here (About 18G).
Dataset Information
Number of samples: 4,516,283 (Keep ~46.77% from the original dataset)
Refining Recipe
#… See the full description on the dataset page: https://huggingface.co/datasets/datajuicer/the-pile-uspto-refined-by-data-juicer.scaling_mia_the_pile_00_FreeLawscaling_mia_the_pile_00_PubMed_Abstractsscaling_mia_the_pile_00_USPTO_Backgroundssmall-the_pilescaling_mia_the_pile_00_EuroParlsmall-the_pile-dedupscaling_mia_the_pile_00_YoutubeSubtitlesscaling_mia_the_pile_00_PhilPapersscaling_mia_the_pile_00_StackExchangethe-pile-europarl-refined-by-data-juicer
The Pile -- EuroParl (refined by Data-Juicer)
A refined version of EuroParl dataset in The Pile by Data-Juicer. Removing some "bad" samples from the original dataset to make it higher-quality.
This dataset is usually used to pretrain a Large Language Model.
Notice: Here is a small subset for previewing. The whole dataset is available here (About 2.2GB).
Dataset Information
Number of samples: 61,601 (Keep ~88.23% from the original dataset)
Refining Recipe… See the full description on the dataset page: https://huggingface.co/datasets/datajuicer/the-pile-europarl-refined-by-data-juicer.