iNeil77/pseudo-mini-pile

A small, aggressively cleaned and de-duped pre-training corpus for academic settings. It aims to recreate something akin to The Pile but prioritizes quality for the constrained token budget academic researchers live with.

It has seven config subsets and an eighth all subset that combines them for a total of ~91B tokens (GPT2 Tokenizer estimate). These splits are as follows:

1. 1.c4_realnews: The RealNews domain subset of the C4 dataset containing news articles.
2. 2.openwebtext: The OpenWebText dataset containing the contents of the links mentioned in Reddit posts with at least 3 upvotes.
3. 3.peS2o: The PeS2o dataset containing academic articles from Semantic Scholar.
4. 4.redpajama_books: The books subset of RedPajama V1.
5. 5.stackexchange: The EN StackExchange non-code subset of the BigScience ROOTs dataset.
6. 6.uspto: The EN USPTO patent applications contents' subset of the BigScience ROOTs dataset.
7. 7.wiki: The EN Wiki subset of the BigScience ROOTs dataset.

The following processing and filtering steps have been applied:

1. 1.Removed citation text and bibliography information for academic texts.
2. 2.Ran a perplexity filter using a KenLM model trained on the EN OSCAR corpus and removed documents with a perplexity of more than 325 and less than 7.
3. 3.Removed samples which have a repeating <=4-gram proportion of 15%.
4. 4.Removed samples which have lower than 99% confidence of being EN using the lingua language detector.
5. 5.Performed an aggressive MinHash de-dupe using a shingle size of 8 and a low threshold of 0.5.