Datasetpublic

emozilla/dolma-v1_7-3B-tokenized-llama3-nanoset

Tokenized (Llama 3) verison of NousResearch/dolma-v1_7-3B as a Nanotron dataset. Can also be used directly with numpy, for example import numpy as np dataset_buffer_mmap = np.memmap("dolma-v1_7-3B-nanoset-l3_input_ids.npy", mode="r", order="C", dtype=np.int32) dataset_buffer = memoryview(dataset_buffer_mmap) dataset_number_of_tokens = int(len(dataset_buffer))

sourceHugging Faceodc-byupdated 2y agoView on Hugging Face

1likes18downloads

Dataset Card

Tokenized (Llama 3) verison of NousResearch/dolma-v1_7-3B as a Nanotron dataset.

Can also be used directly with numpy, for example

python

import numpy as np

dataset_buffer_mmap = np.memmap("dolma-v1_7-3B-nanoset-l3_input_ids.npy", mode="r", order="C", dtype=np.int32)
dataset_buffer = memoryview(dataset_buffer_mmap)
dataset_number_of_tokens = int(len(dataset_buffer))