Datasetpublic

malaysia-ai/mosaic-starcoder-filtered

Mosaic format for filtered starcoder dataset to train Malaysian LLM This repository is to store dataset shards using mosaic format. prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-starcoder.ipynb using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer 4096 context length. how-to git clone, git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-starcoder-filtered load it, from streaming import… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/mosaic-starcoder-filtered.

sourceHugging Faceupdated 3y agoView on Hugging Face

0likes3.9kdownloads

Dataset Card

Mosaic format for filtered starcoder dataset to train Malaysian LLM

This repository is to store dataset shards using mosaic format.

1.prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-starcoder.ipynb
2.using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer
3.4096 context length.

how-to

1.git clone,

bash

git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-starcoder-filtered

1.load it,

python

from streaming import LocalDataset
import numpy as np
from streaming.base.format.mds.encodings import Encoding, _encodings

class UInt16(Encoding):
    def encode(self, obj) -> bytes:
        return obj.tobytes()

    def decode(self, data: bytes):
        return np.frombuffer(data, np.uint16)

_encodings['uint16'] = UInt16

dataset = LocalDataset('mosaic-starcoder-filtered')
len(dataset)