Natooka/parameter-golf-sp-tokenizers

Parameter Golf SP16384 — Tokenizer + Tokenized FineWeb-10B Shards

SentencePiece BPE tokenizer (vocab_size=16384, byte_fallback=True) + the full FineWeb-10B corpus pre-tokenized with it. Companion artifact to the chaoscontrol submission pipeline; published to make submission-day setup frictionless — no corpus download, no re-tokenization.

Files

Tokenizer (root)

• —fineweb_16384_bpe.model — SentencePiece model (455 KB).
• —fineweb_16384_bpe.vocab — Human-readable vocab sidecar (185 KB).

Tokenized shards (shards/)

• —shards/fineweb_val_000000.bin — val shard, 42,266,034 tokens (~84 MB, uint16).
• —shards/fineweb_train_{000000..000132}.bin — 133 train shards, 13,262,831,920 tokens (~25 GB total, uint16).

Shards are flat uint16 little-endian token streams, no header, concatenated docs with no separators. Each shard ends on a doc boundary. First 50k docs of docs_selected.jsonl → val, rest → train (file order, per the upstream manifest contract).

Submission-day usage

Pull the whole thing in ~5 min on a typical pod:

from huggingface_hub import snapshot_download

local = snapshot_download(
    repo_id="Natooka/parameter-golf-sp-tokenizers",
    repo_type="dataset",
    local_dir="baselines/parameter_golf",
    # Optional — get just shards, or just the model:
    # allow_patterns=["shards/*.bin", "fineweb_*.model"],
)

After the download you have baselines/parameter_golf/shards/*.bin + baselines/parameter_golf/fineweb_16384_bpe.model. Point your training runner at those paths.

Training configuration (tokenizer)

Training was single-threaded in the BPE merge phase (SP 0.2.1 limitation — num_threads helps normalization only). Wall-clock on 28 vCPU Xeon 8480+ host: ~25 min SP training + ~30 min full-corpus tokenization.

Reproducing from scratch

python scripts/build_sp_shards.py \
  --docs-path path/to/docs_selected.jsonl \
  --vocab-size 16384 \
  --sp-seed 1337 \
  --sp-train-docs 5000000 \
  --num-workers 28

Byte-identical outputs are guaranteed within a matching (vocab_size, sp_seed, num_workers) triple — SP's multi-threaded merge counting can drift on tie-breaks across thread counts. Use the same --num-workers for cross-machine determinism, or pin to --num-workers 1 for strict identity.

License

CC-BY 4.0 for our artifacts (tokenizer + pre-tokenized shards). Upstream docs_selected.jsonl subject to the Parameter Golf competition's terms (from willdepueoai/parameter-golf).