Datasetpublic

something-else/Slim-binidx

This dataset comprises nine chunks (out of ten) from the Cerebras/SlimPajama-627B dataset, processed into a binary index (bin idx) format. The first chunk is located at : rwkv-x-dev/slimpajama-binidx. Due to their large size, each chunk is split into multiple parts for easier handling. To reassemble and decompress these parts, follow these steps: Combine all parts of the desired chunk into a single file: cat chunk2_text_document_part_* > chunk2_text_document.tar.xz Decompress the combined… See the full description on the dataset page: https://huggingface.co/datasets/something-else/Slim-binidx.

sourceHugging Faceupdated 3y agoView on Hugging Face

1likes989downloads

Dataset Card

This dataset comprises nine chunks (out of ten) from the ``Cerebras/SlimPajama-627B` dataset, processed into a binary index (bin idx) format. The first chunk is located at : `rwkv-x-dev/slimpajama-binidx``.

Due to their large size, each chunk is split into multiple parts for easier handling. To reassemble and decompress these parts, follow these steps:

1.Combine all parts of the desired chunk into a single file:

cat chunk2_text_document_part_* > chunk2_text_document.tar.xz```


2. Decompress the combined file:

1.Extract the decompressed file:

tar -xvf chunk2_text_document.tar```


This process successfully reassembles and decompresses the chosen chunk.

cat chunk2textdocumentpart* > chunk2textdocument.tar.xz xz -d chunk2textdocument.tar.xz tar -xvf chunk2textdocument.tar