Datasetpublic

malaysia-ai/mosaic-dedup-text-dataset

Mosaic format for dedup text dataset to train Malaysian LLM This repository is to store dataset shards using mosaic format. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-dedup-text-dataset-4096.ipynb using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer 4096 context length. how-to git clone, git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset load it, from streaming… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset.

sourceHugging Faceupdated 3y agoView on Hugging Face

0likes2.9kdownloads

Dataset Card

Mosaic format for dedup text dataset to train Malaysian LLM

This repository is to store dataset shards using mosaic format.

1.prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-dedup-text-dataset-4096.ipynb
2.using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer
3.4096 context length.

how-to

1.git clone,

bash

git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-dedup-text-dataset

1.load it,

python

from streaming import LocalDataset
import numpy as np
from streaming.base.format.mds.encodings import Encoding, _encodings

class UInt16(Encoding):
    def encode(self, obj) -> bytes:
        return obj.tobytes()

    def decode(self, data: bytes):
        return np.frombuffer(data, np.uint16)

_encodings['uint16'] = UInt16

dataset = LocalDataset('mosaic-dedup-text-dataset')
len(dataset)