LightSpeedUp/parameter-golf-data
Parameter Golf Competition Data Pre-tokenized FineWeb shards for the OpenAI Parameter Golf competition. Multiple SentencePiece vocab sizes plus a corrected byte-exact Scylla (TokenMonster) tokenization. Free checkpoint persistence API. Zero setup friction. ⚠️ Important: Scylla v1 Deprecated The original fineweb_scylla/ directory uses the 998-token vocab from PR #1143. That vocab's byte-accounting metadata treated TokenMonster tokens as context-free, which… See the full description on the dataset page: https://huggingface.co/datasets/LightSpeedUp/parameter-golf-data.
Update dataset card for v2: Scylla v2 + SP4096/8192/12288/16384 (#4)
Add fineweb_sp16384/ (SentencePiece BPE 16384) (fineweb_train_000000.bin) (#9)
Add fineweb_sp12288/ (SentencePiece BPE 12288) (fineweb_train_000000.bin) (#8)
Add fineweb_byte260/ (pure-byte PureByteTokenizer pad=256 bos=257) (fineweb_train_000000.bin) (#7)
Add fineweb_sp4096/ (SentencePiece BPE 4096) (fineweb_train_000000.bin) (#6)
Add fineweb_sp8192/ (SentencePiece BPE 8192) (fineweb_train_000000.bin) (#5)
Add fineweb_scylla_v2/ (corrected byte-exact 1254-token vocab, PR #1314) (#3)
Add tokenizers/scylla_v2/scylla.meta.npz (PR #1314 byte-exact Scylla) (#2)
Add tokenizers/scylla_v2/scylla.yaml (PR #1314 byte-exact Scylla) (#1)
Upload folder using huggingface_hub
Upload folder using huggingface_hub
Upload README.md with huggingface_hub
Upload folder using huggingface_hub
Upload folder using huggingface_hub
Upload SHA256SUMS.txt with huggingface_hub
Upload PATENTS.md with huggingface_hub
Upload README.md with huggingface_hub
initial commit