LightSpeedUp/parameter-golf-data
Parameter Golf Competition Data Pre-tokenized FineWeb shards for the OpenAI Parameter Golf competition. Multiple SentencePiece vocab sizes plus a corrected byte-exact Scylla (TokenMonster) tokenization. Free checkpoint persistence API. Zero setup friction. ⚠️ Important: Scylla v1 Deprecated The original fineweb_scylla/ directory uses the 998-token vocab from PR #1143. That vocab's byte-accounting metadata treated TokenMonster tokens as context-free, which… See the full description on the dataset page: https://huggingface.co/datasets/LightSpeedUp/parameter-golf-data.
0692
1# Patent Boundary Notice2 3## Scope4 5The retokenization pipeline (`retokenize_scylla.py`) included in this repository converts6SentencePiece-tokenized FineWeb shards to alternative tokenizer vocabularies. The algorithm7is straightforward: decode tokens to bytes, re-encode with the target tokenizer, validate8byte-level roundtrip fidelity.9 10## What is NOT covered11 12The pre-tokenized data files (`.bin` shards) in this dataset are standard binary token13sequences. They do not embody any patentable method — they are the output of running a14tokenizer on public data.15 16## What MAY be covered17 18Techniques described in patent applications by Light Speed Up LLC related to:19- Sensitivity-guided mixed-precision quantization20- Adaptive tokenizer selection for language model compression21 22These techniques are implemented in training code (not in this dataset or retokenization tool).23The Apache 2.0 license on the code grants patent rights for the code as-shipped.24 25## Contact26 27For patent-related inquiries: mato@lightspeedup.com28