Datasetpublic

emozilla/dolma-v1_7-3B-tokenized-llama3-nanoset

Tokenized (Llama 3) verison of NousResearch/dolma-v1_7-3B as a Nanotron dataset. Can also be used directly with numpy, for example import numpy as np dataset_buffer_mmap = np.memmap("dolma-v1_7-3B-nanoset-l3_input_ids.npy", mode="r", order="C", dtype=np.int32) dataset_buffer = memoryview(dataset_buffer_mmap) dataset_number_of_tokens = int(len(dataset_buffer))

sourceHugging Faceodc-byupdated 2y agoView on Hugging Face

1likes18downloads

settings

This repository belongs to emozilla on Hugging Face.

CoolFace never edits a repository it does not host. Visibility, licence, collaborators and gating are all managed at the source.

namedolma-v1_7-3B-tokenized-llama3-nanoset

visibilitypublic

licenceodc-by

gatedno

owneremozilla

Account settings