CoolFace
Datasetpublic

gyung/gdn2-cpt-fineweb-edu-30k

gdn2-370m CPT pack — gdn2-cpt-fineweb-edu-30k Pretraining-corpus pack used for the GDN-2 370M SSKetch+ReMoE 4-way CPT runs (2026-08-23). Source & attribution Origin dataset: HuggingFaceFW/fineweb-edu config='sample-100BT' (public; used under the terms stated on its dataset card) Sampling recipe: streaming skip=800,000, num_samples=30,000 documents Tokenizer: TinyLlama/TinyLlama_v1.1 Total: 33,569,012 tokens across 30,000 documents Contents… See the full description on the dataset page: https://huggingface.co/datasets/gyung/gdn2-cpt-fineweb-edu-30k.

sourceHugging Faceupdated 1mo agoView on Hugging Face
0likes27downloads
Dataset Card

gdn2-370m CPT pack — gdn2-cpt-fineweb-edu-30k

Pretraining-corpus pack used for the GDN-2 370M SSKetch+ReMoE 4-way CPT runs (2026-08-23).

Source & attribution

  • —Origin dataset: `HuggingFaceFW/fineweb-edu` config='sample-100BT' (public; used under the terms stated on its dataset card)
  • —Sampling recipe: streaming skip=800,000, num_samples=30,000 documents
  • —Tokenizer: `TinyLlama/TinyLlama_v1.1`
  • —Total: 33,569,012 tokens across 30,000 documents

Contents

splitcolumndescription
packedinput_idsthe exact 8,195 contiguous 4096-token blocks fed to training
documentstextthe detokenized source documents, in sampling order

Intended use

Continued-pretraining research for linear-recurrent LMs (Gated DeltaNet v2, Memory-Caching SSC, SSKetch routing, ReMoE gating). Redistribution keeps the origin dataset's license; attribution to the original authors is given above.