Natooka/parameter-golf-sp-tokenizers
Parameter Golf SP16384 — Tokenizer + Tokenized FineWeb-10B Shards SentencePiece BPE tokenizer (vocab_size=16384, byte_fallback=True) + the full FineWeb-10B corpus pre-tokenized with it. Companion artifact to the chaoscontrol submission pipeline; published to make submission-day setup frictionless — no corpus download, no re-tokenization. Files Tokenizer (root) fineweb_16384_bpe.model — SentencePiece model (455 KB). fineweb_16384_bpe.vocab —… See the full description on the dataset page: https://huggingface.co/datasets/Natooka/parameter-golf-sp-tokenizers.
Conversations for this repository live on Hugging Face.
CoolFace shows imported repositories read-only. Posting into someone else’s repository from here would need an authorised integration and the account holder’s consent, so the link goes to the source instead.
Open discussions on Hugging Face