Datasetpublic

emozilla/dolma-v1_7-30B-tokenized-llama2-nanoset

Tokenized (Llama 2) verison of emozilla/dolma-v1_7-30B as a Nanotron dataset split into 10 GB chunks. To download: huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-30B-tokenized-llama2-nanoset --local-dir-use-symlinks False emozilla/dolma-v1_7-30B-tokenized-llama2-nanoset To recombine: cat dolma-v1_7-30B-tokenized-llama2-nanoset/dolma-v1_7-30B-tokenized-llama2-nanoset_input_ids.npy.* > dolma-v1_7-30B-tokenized-llama2-nanoset.npy rm -rf… See the full description on the dataset page: https://huggingface.co/datasets/emozilla/dolma-v1_7-30B-tokenized-llama2-nanoset.

sourceHugging Faceodc-byupdated 2y agoView on Hugging Face

1likes108downloads

Dataset Card

Tokenized (Llama 2) verison of emozilla/dolma-v1_7-30B as a Nanotron dataset split into 10 GB chunks.

To download:

shell

huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-30B-tokenized-llama2-nanoset --local-dir-use-symlinks False emozilla/dolma-v1_7-30B-tokenized-llama2-nanoset

To recombine:

shell

cat dolma-v1_7-30B-tokenized-llama2-nanoset/dolma-v1_7-30B-tokenized-llama2-nanoset_input_ids.npy.* > dolma-v1_7-30B-tokenized-llama2-nanoset.npy
rm -rf dolma-v1_7-30B-tokenized-llama2-nanoset

Can also be used directly with numpy, for example

python

import numpy as np

dataset_buffer_mmap = np.memmap("dolma-v1_7-30B-tokenized-llama2-nanoset.npy",
  mode="r", order="C", dtype=np.int16)
dataset_buffer = memoryview(dataset_buffer_mmap)
dataset_number_of_tokens = int(len(dataset_buffer))