Datasetpublic

AlppAI/SlimPajama-chunked

SlimPajama-Chunked Dataset Description This is a chunked re-upload of Cerebras' SlimPajama-627B. The original upload has split the dataset into 10 chunks, with each containing upwards of 5,000 files. This makes it cumbersome to download and process. We've downloaded the entire dataset for our own purposes, and decided to upload the chunked version for easier usage. Each file is ~45GB due to HuggingFace's limitation of 50GB per LFS file.

sourceHugging Faceupdated 3y agoView on Hugging Face

5likes642downloads

Dataset Card

SlimPajama-Chunked

Dataset Description

This is a chunked re-upload of Cerebras' SlimPajama-627B. The original upload has split the dataset into 10 chunks, with each containing upwards of 5,000 files. This makes it cumbersome to download and process. We've downloaded the entire dataset for our own purposes, and decided to upload the chunked version for easier usage.

Each file is ~45GB due to HuggingFace's limitation of 50GB per LFS file.