CoolFace
Datasetpublic

NAME0x0/hagi-fineweb-edu-smollm2

HAGI - Tokenized HuggingFaceFW/fineweb-edu (SmolLM2-135M tokenizer) Pre-tokenized token-id shards used to train the HAGI Stage 0 baseline and the Grade-Decomposed Recurrence ablation (models A/B/C/D). A tokenized derivative of HuggingFaceFW/fineweb-edu subset sample-10BT, published so the exact training corpus loads identically in any environment (Colab, Kaggle, local) with no re-tokenization and no Google Drive access. Format - read before using Files:… See the full description on the dataset page: https://huggingface.co/datasets/NAME0x0/hagi-fineweb-edu-smollm2.

sourceHugging Faceodc-byupdated 4mo agoView on Hugging Face
0likes38downloads
Dataset Card

HAGI - Tokenized HuggingFaceFW/fineweb-edu (SmolLM2-135M tokenizer)

Pre-tokenized token-id shards used to train the HAGI Stage 0 baseline and the Grade-Decomposed Recurrence ablation (models A/B/C/D). A tokenized derivative of `HuggingFaceFW/fineweb-edu` subset sample-10BT, published so the exact training corpus loads identically in any environment (Colab, Kaggle, local) with no re-tokenization and no Google Drive access.

Format - read before using

  • —Files: shard_NNNNN.bin - a flat little-endian `uint16` stream of token ids, no header, documents separated by the tokenizer's EOS id. Read via numpy.memmap.
  • —Tokenizer: `HuggingFaceTB/SmolLM2-135M` - vocab 49,152 (fits uint16). Shards tokenized with any other tokenizer are incompatible: the ids would index the wrong embeddings (silent garbage), so do not mix sources.
  • —Token count of a shard: filesize / 2.
  • —Held-out convention: the last shard (shard_00006.bin) is reserved as validation; train on the rest.

Contents

  • —Shards: 7
  • —Total tokens: ~630,891,027
ShardTokens
shard_00000.bin100,694,996
shard_00001.bin100,686,108
shard_00002.bin100,677,968
shard_00003.bin100,324,700
shard_00004.bin100,641,381
shard_00005.bin100,534,502
shard_00006.bin27,331,372

Load

python
from huggingface_hub import snapshot_download
path = snapshot_download(repo_id="NAME0x0/hagi-fineweb-edu-smollm2", repo_type="dataset", allow_patterns="*.bin")
# point training at the returned directory:
#   python -m prototype.training.train --config configs/ablation_b.yaml --data <path> ...

MemmapTokenDataset (prototype/data/dataset.py) consumes this directory directly.

Models trained on this data

  • —Stage 0 baseline: https://huggingface.co/NAME0x0/hagi-stage0
  • —Ablation: `-a` / `-b` / `-c` / `-d`

License & attribution

Derivative of HuggingFaceFW/fineweb-edu (sample-10BT), released under ODC-By 1.0 - the upstream license. Attribute FineWeb-Edu (HuggingFaceFW) on use. Tokenization adds no new content; it only maps text to HuggingFaceTB/SmolLM2-135M ids.