zee-drytis/damr-zyda2-64k-tokenized
DAMR Zyda2 64K Tokenized This repository contains a deterministic tokenized representation of a 1,749,978,785,641-token subset of Zyda-2 for language-model pretraining. Format Files under data/ contain contiguous little-endian uint16 token IDs. Concatenate shards in numeric order to reproduce the original stream. The shared 64K BPE tokenizer is stored under tokenizer/tokenizer.json. Shard manifests provide byte offsets, token counts, and SHA-256 checksums. The… See the full description on the dataset page: https://huggingface.co/datasets/zee-drytis/damr-zyda2-64k-tokenized.
Add train shard manifest
Upload data/train-00814-of-00815.uint16.bin
Upload data/train-00813-of-00815.uint16.bin
Upload data/train-00812-of-00815.uint16.bin
Upload data/train-00811-of-00815.uint16.bin
Upload data/train-00810-of-00815.uint16.bin
Upload data/train-00809-of-00815.uint16.bin
Upload data/train-00808-of-00815.uint16.bin
Upload data/train-00807-of-00815.uint16.bin
Upload data/train-00806-of-00815.uint16.bin
Upload data/train-00805-of-00815.uint16.bin
Upload data/train-00804-of-00815.uint16.bin
Upload data/train-00803-of-00815.uint16.bin
Upload data/train-00802-of-00815.uint16.bin
Upload data/train-00801-of-00815.uint16.bin
Upload data/train-00800-of-00815.uint16.bin
Upload data/train-00799-of-00815.uint16.bin
Upload data/train-00798-of-00815.uint16.bin
Upload data/train-00797-of-00815.uint16.bin
Upload data/train-00796-of-00815.uint16.bin
Upload data/train-00795-of-00815.uint16.bin
Upload data/train-00794-of-00815.uint16.bin
Upload data/train-00793-of-00815.uint16.bin
Upload data/train-00792-of-00815.uint16.bin
Upload data/train-00791-of-00815.uint16.bin
Upload data/train-00790-of-00815.uint16.bin
Upload data/train-00789-of-00815.uint16.bin
Upload data/train-00788-of-00815.uint16.bin
Upload data/train-00787-of-00815.uint16.bin
Upload data/train-00786-of-00815.uint16.bin
Upload data/train-00785-of-00815.uint16.bin
Upload data/train-00784-of-00815.uint16.bin
Upload data/train-00783-of-00815.uint16.bin
Upload data/train-00782-of-00815.uint16.bin
Upload data/train-00781-of-00815.uint16.bin
Upload data/train-00780-of-00815.uint16.bin
Upload data/train-00779-of-00815.uint16.bin
Upload data/train-00778-of-00815.uint16.bin
Upload data/train-00777-of-00815.uint16.bin
Upload data/train-00776-of-00815.uint16.bin
Upload data/train-00775-of-00815.uint16.bin
Upload data/train-00774-of-00815.uint16.bin
Upload data/train-00773-of-00815.uint16.bin
Upload data/train-00772-of-00815.uint16.bin
Upload data/train-00771-of-00815.uint16.bin
Upload data/train-00770-of-00815.uint16.bin
Upload data/train-00769-of-00815.uint16.bin
Upload data/train-00768-of-00815.uint16.bin
Upload data/train-00767-of-00815.uint16.bin
Upload data/train-00766-of-00815.uint16.bin