JeanKaddour/minipile
Dataset Card for MiniPile Dataset Description The MiniPile Challenge for Data-Efficient Language Models Dataset Summary MiniPile is a 6GB subset of the deduplicated The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters. The primary motivation for curating MiniPile is that… See the full description on the dataset page: https://huggingface.co/datasets/JeanKaddour/minipile.
Update README.md
Upload README.md with huggingface_hub
Upload data/test-00000-of-00001-010a6231c4b54d31.parquet with huggingface_hub
Upload data/validation-00000-of-00001-a2192e61a091cecb.parquet with huggingface_hub
Upload data/train-00011-of-00012-aec474909333c631.parquet with huggingface_hub
Upload data/train-00010-of-00012-d266658ccbfa0537.parquet with huggingface_hub
Upload data/train-00009-of-00012-0b640f47936d940a.parquet with huggingface_hub
Upload data/train-00008-of-00012-3273ba93936ad8ef.parquet with huggingface_hub
Upload data/train-00007-of-00012-239b43e016d4ac92.parquet with huggingface_hub
Upload data/train-00006-of-00012-89040916c30140e6.parquet with huggingface_hub
Upload data/train-00005-of-00012-d255c96cd87a0aa7.parquet with huggingface_hub
Upload data/train-00004-of-00012-a6a94a0207e8e96c.parquet with huggingface_hub
Upload data/train-00003-of-00012-47006e5a888a9324.parquet with huggingface_hub
Upload data/train-00002-of-00012-efb6c8de04272068.parquet with huggingface_hub
Upload data/train-00001-of-00012-2bb9d088068a84c9.parquet with huggingface_hub
Upload data/train-00000-of-00012-6fbcb5acda05b3c0.parquet with huggingface_hub
initial commit