Datasetpublic

tachytelicdetonation/ncp-1p6b-mix

ncp-1p6b-mix Packed uint16 token stream for NCP-64M pretraining (1,601,363,502 tokens). Mix (token-weighted, deficit-scheduled interleave) finepdfs (codelion/finepdfs-1B): 45.5% — 729M tok dclm (codelion/dclm-baseline-1B): 26.8% — 428M tok fineweb_edu (codelion/fineweb-edu-1B): 18.0% — 287M tok sutra (codelion/sutra-improved-100M): 10.1% — 162M tok Backbone follows the 70M-scale recipe from https://huggingface.co/blog/codelion/optimal-dataset-mixing (50/30/20… See the full description on the dataset page: https://huggingface.co/datasets/tachytelicdetonation/ncp-1p6b-mix.

sourceHugging Faceapache-2.0updated 13d agoView on Hugging Face

0likes31downloads

Dataset Card

ncp-1p6b-mix

Packed uint16 token stream for NCP-64M pretraining (1,601,363,502 tokens).

Mix (token-weighted, deficit-scheduled interleave)

—finepdfs (codelion/finepdfs-1B): 45.5% — 729M tok
—dclm (codelion/dclm-baseline-1B): 26.8% — 428M tok
—fineweb_edu (codelion/fineweb-edu-1B): 18.0% — 287M tok
—sutra (codelion/sutra-improved-100M): 10.1% — 162M tok

Backbone follows the 70M-scale recipe from https://huggingface.co/blog/codelion/optimal-dataset-mixing (50/30/20 finepdfs/DCLM/FineWeb-Edu) with 10% Sutra pedagogical data.

Val bins (val/)

Per-source holdouts (~400k tok each) + ood_finewiki.bin (504k tok, HuggingFaceFW/finewiki@en — never in the train stream).

Format

Flat uint16 tokens, EOS-separated docs, MiniMind 6400-vocab BPE tokenizer. Read: np.memmap(path, dtype=np.uint16).

Reproduce

scripts/prepare_data.py in github.com/tachytelicdetonation/NCP-64M, seed 0, with the --source/--ood specs above (see repo README).