LightSpeedUp/parameter-golf-data

1---2license: odc-by3task_categories:4  - text-generation5language:6  - en7pretty_name: Parameter Golf Competition Data v28size_categories:9  - 1B<n<10B10tags:11  - parameter-golf12  - fineweb13  - language-modeling14  - competition15---16 17# Parameter Golf Competition Data18 19Pre-tokenized [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) shards for the [OpenAI Parameter Golf](https://github.com/openai/parameter-golf) competition. Multiple SentencePiece vocab sizes plus a corrected byte-exact Scylla (TokenMonster) tokenization. Free checkpoint persistence API. Zero setup friction.20 21---22 23## ⚠️ Important: Scylla v1 Deprecated24 25The original `fineweb_scylla/` directory uses the 998-token vocab from [PR #1143](https://github.com/openai/parameter-golf/pull/1143). That vocab's byte-accounting metadata treated TokenMonster tokens as context-free, which overcounts source bytes by ~4%. Any `val_bpb` reported through the standard pipeline on `fineweb_scylla/` is inflated by roughly the same factor.26 27The bug is tracked in [Issue #897](https://github.com/openai/parameter-golf/issues/897) and corrected in [PR #1314](https://github.com/openai/parameter-golf/pull/1314) (simon-marcus, "Scylla: Corrected Byte-Exact Tokenizer Path"). The corrected path uses a full byte-native TokenMonster regime (`charset=none`, `capcode=0`, `normalization=none`, explicit 0x00–0xFF byte fallback) and is byte-exact on the fixed FineWeb validation text.28 29**Use `fineweb_scylla_v2/` for any new work.** Old `fineweb_scylla/` is kept for reproducibility of past runs and will not be deleted, but should not be used for new BPB comparisons.30 31---32 33## How It Works (The Simple Version)34 35Think of it like plumbing:36 371. **The reservoir** is this dataset — competition data, pre-processed and ready to flow.382. **The pipe** is `huggingface-cli download` — one command, and data flows to your GPU pod. Fast, resumable. If the pipe breaks mid-transfer, reconnect and it picks up where it left off.393. **Your pod** is the sink — data arrives at `/workspace/data/`, ready to use. No processing, no conversion, no waiting.404. **The safety valve** is checkpoint persistence — every N steps, your training progress flows out to cloud storage. Pod dies? New pod picks up the flow from the last save. No lost work.41 42That's it. Data flows in. Checkpoints flow out. You train in between.43 44### Step by Step45 46**I just want to train. What do I do?**47 48```bash49# Step 1: Install huggingface-cli (if you don't have it)50pip install huggingface-hub51 52# Step 2: Download the competition data (SP1024 default)53huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp1024/*" --local-dir /workspace/data54 55# Step 3: That's it. Train.56python train_gpt.py --data_dir /workspace/data/fineweb_sp102457```58 59**I want a bigger vocab:**60 61```bash62# SP4096 — good middle ground63huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp4096/*" --local-dir /workspace/data64 65# SP8192 — a step up in vocab capacity66huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp8192/*" --local-dir /workspace/data67```68 69**I want to save checkpoints so I don't lose work:**70 71```bash72# After every N steps in your training script, save:73curl -X PUT \74  -H "Authorization: Bearer YOUR_GITHUB_TOKEN" \75  --data-binary @checkpoint.pt \76  https://pgolf-api.lightspeedup.com/put/YOUR_GITHUB_USERNAME/my-run/checkpoint.pt77 78# On a new pod, resume:79curl -o checkpoint.pt \80  -H "Authorization: Bearer YOUR_GITHUB_TOKEN" \81  "https://pgolf-api.lightspeedup.com/download?run_id=my-run&filename=checkpoint.pt"82```83 84**I want the fully automated experience:**85 86Use our RunPod template: `matotezitanka/proteus-pytorch:community`87 88Set these env vars before launch:89- `PGOLF_DATA=sp1024` (or `sp4096`, `sp8192`, `sp12288`, `sp16384`, or `scylla_v2`)90- `PGOLF_SHARDS=full` (or `mini` for testing)91- `PGOLF_GITHUB_TOKEN=ghp_yourtoken` (optional, for checkpoints)92- `PGOLF_USER=yourgithubname` (optional)93- `PGOLF_RUN=my-experiment-1` (optional)94 95Hit deploy. SSH in when it's ready. Everything is there.96 97---98 99## Comprehensive Guide100 101### Available Data102 103| Tokenizer | Vocab | Size | Use case |104|-----------|-------|------|----------|105| **SP1024** | 1024 tokens | ~15 GB | Competition default. Most PRs use this. |106| **SP4096** | 4096 tokens | ~12 GB | Larger vocab, shorter sequences per doc. |107| **SP8192** | 8192 tokens | ~11 GB | Common for bigram/mixer submissions. |108| **SP12288** | 12288 tokens | ~10 GB | Explores the 8k–16k vocab range. |109| **SP16384** | 16384 tokens | ~9 GB | Largest SentencePiece variant we publish. |110| **byte260** | 260 tokens | ~40 GB | Pure-byte tokenization. Bytes `0x00..0xFF` → ids `0..255` directly. Reserved specials: `pad=256, bos=257, eos=258, unk=259`. Encoding is `[257] + list(text.encode("utf-8"))`. No SentencePiece model involved. ~195 train + 2 val shards (byte density is ~4× SP1024). |111| **Scylla v2** | 1254 tokens | ~17 GB | **Corrected** TokenMonster byte-exact tokenizer (PR #1314). Leaderboard-comparable BPB. |112| ~~Scylla v1~~ | ~~998 tokens~~ | ~~~11 GB~~ | **Deprecated** — buggy byte accounting (Issue #897). Kept for reproducibility only. |113 114Each directory contains 80 training shards + 1 validation shard + tokenizer models.115 116### Download Options117 118```bash119# === DATA SELECTION ===120 121# SP1024 — competition default (~15 GB)122huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp1024/*" --local-dir /workspace/data123 124# SP4096 — 4k vocab (~12 GB)125huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp4096/*" --local-dir /workspace/data126 127# SP8192 — 8k vocab (~11 GB)128huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp8192/*" --local-dir /workspace/data129 130# SP12288 — 12k vocab (~10 GB)131huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp12288/*" --local-dir /workspace/data132 133# SP16384 — 16k vocab (~9 GB)134huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp16384/*" --local-dir /workspace/data135 136# byte260 — UTF-8 bytes + 4 reserved specials, no tokenizer training (~40 GB)137huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_byte260/*" --local-dir /workspace/data138 139# Scylla v2 — corrected TokenMonster 1254-token vocab (~17 GB)140huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_scylla_v2/*" --local-dir /workspace/data141 142# Legacy Scylla v1 — 998-token vocab, deprecated (use v2 instead)143huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_scylla/*" --local-dir /workspace/data144 145# All datasets + all tokenizers (~70 GB)146huggingface-cli download LightSpeedUp/parameter-golf-data --local-dir /workspace/data147 148# Just tokenizer models (tiny, < 2 MB)149huggingface-cli download LightSpeedUp/parameter-golf-data --include "tokenizers/*" --local-dir /workspace/data150 151 152# === SHARD SUBSETS (save time/bandwidth) ===153 154# Mini — 10 shards for smoke tests (~2 GB)155huggingface-cli download LightSpeedUp/parameter-golf-data \156    --include "fineweb_sp1024/fineweb_train_00000?.bin" \157    --include "fineweb_sp1024/fineweb_val*" \158    --local-dir /workspace/data159 160# Half — 40 shards (~7 GB)161huggingface-cli download LightSpeedUp/parameter-golf-data \162    --include "fineweb_sp1024/fineweb_train_0000[0-3]?.bin" \163    --include "fineweb_sp1024/fineweb_val*" \164    --local-dir /workspace/data165 166# Val only — just validation data (~200 MB)167huggingface-cli download LightSpeedUp/parameter-golf-data \168    --include "fineweb_sp1024/fineweb_val*" \169    --local-dir /workspace/data170```171 172**No HuggingFace account required.** This is a public dataset. No login, no token, no signup.173 174**Downloads are resumable.** If your connection drops, re-run the same command and it picks up where it left off.175 176### Checkpoint Persistence API177 178Save and resume training across pod preemptions. Your GitHub token is your identity — no accounts to create.179 180```bash181# === CHECKPOINT API (https://pgolf-api.lightspeedup.com) ===182 183# Upload a checkpoint184curl -X POST https://pgolf-api.lightspeedup.com/upload \185  -H "Authorization: Bearer ghp_yourtoken" \186  -H "Content-Type: application/json" \187  -d '{"run_id": "my-run", "filename": "checkpoint_step500.pt"}'188 189# Then PUT the file190curl -X PUT https://pgolf-api.lightspeedup.com/put/yourusername/my-run/checkpoint_step500.pt \191  -H "Authorization: Bearer ghp_yourtoken" \192  --data-binary @checkpoint_step500.pt193 194# Download a checkpoint195curl -o checkpoint.pt \196  -H "Authorization: Bearer ghp_yourtoken" \197  "https://pgolf-api.lightspeedup.com/download?run_id=my-run&filename=checkpoint_step500.pt"198 199# List your checkpoints200curl -H "Authorization: Bearer ghp_yourtoken" \201  "https://pgolf-api.lightspeedup.com/list?run_id=my-run"202 203# Delete a run's checkpoints204curl -X DELETE -H "Authorization: Bearer ghp_yourtoken" \205  "https://pgolf-api.lightspeedup.com/clean?run_id=my-run"206```207 208**Limits:** 10 checkpoints per user, 2 GB max each. Auto-deleted after 7 days.209 210### Docker Image211 212```bash213docker pull matotezitanka/proteus-pytorch:community214```215 216Includes: PyTorch 2.11.0 + CUDA 12.8 + Flash Attention 3 + all competition deps (brotli, tokenmonster, sentencepiece) + tools (cpu_test.py, retokenizer, swap_pytorch.sh) + automated boot script.217 218Works on RunPod, Vast.ai, or any Docker host with NVIDIA GPUs.219 220### Data Integrity221 222Every file has a SHA256 checksum. After downloading:223 224```bash225cd /workspace/data226sha256sum -c SHA256SUMS.txt227```228 229If any checksum fails, re-download that file. The download is resumable — you won't re-download files that are already correct.230 231### Dataset Structure232 233```234parameter-golf-data/235├── fineweb_sp1024/        # 80 train + 1 val, SentencePiece BPE 1024236├── fineweb_sp4096/        # 80 train + 1 val, SentencePiece BPE 4096237├── fineweb_sp8192/        # 80 train + 1 val, SentencePiece BPE 8192238├── fineweb_sp12288/       # 80 train + 1 val, SentencePiece BPE 12288239├── fineweb_sp16384/       # 80 train + 1 val, SentencePiece BPE 16384240├── fineweb_byte260/       # ~195 train + 2 val shards, pure-byte (PureByteTokenizer: pad=0, bos=1, eos=2, unk=3, bytes 4..259)241├── fineweb_scylla_v2/     # 80 train + 1 val, corrected TokenMonster 1254-token (PR #1314)242├── fineweb_scylla/        # DEPRECATED — 998-token buggy vocab, kept for reproducibility243├── tokenizers/244│   ├── fineweb_1024_bpe.model           (SP1024 SentencePiece)245│   ├── fineweb_4096_bpe.model           (SP4096 SentencePiece)246│   ├── fineweb_8192_bpe.model           (SP8192 SentencePiece)247│   ├── fineweb_12288_bpe.model          (SP12288 SentencePiece)248│   ├── fineweb_16384_bpe.model          (SP16384 SentencePiece)249│   ├── scylla_v2/scylla.yaml            (TokenMonster 1254, corrected)250│   ├── scylla_v2/scylla.meta.npz        (byte LUTs, byte-exact)251│   ├── scylla/candidate.vocab           (TokenMonster 998, deprecated)252│   └── scylla/candidate.meta.npz        (old byte LUTs, overcounts)253├── SHA256SUMS.txt254└── PATENTS.md255```256 257---258 259## Security & Privacy260 261We believe in transparency. Here's exactly what we can and can't see.262 263### What we CAN access (technically)264- **Your checkpoint files** — they're stored in our Cloudflare R2 bucket. We have admin access to the bucket. We don't look at them, but we could.265- **Your checkpoint metadata** — filenames, sizes, upload timestamps. This is visible in the R2 dashboard.266- **Request logs** — Cloudflare logs request metadata (IP addresses, timestamps, URLs) by default. We do not add any additional logging.267- **Your GitHub username** — extracted from your token to scope your storage namespace.268 269### What we CANNOT access270- **Your training code** — it runs on your pod, never touches our infrastructure.271- **Your model weights** (unless you upload them as a checkpoint).272- **Your GitHub token contents** — the token transits our Worker to call GitHub's API, but it is NOT stored, NOT logged, and NOT persisted anywhere. It's used once per request for authentication and discarded.273- **Other users' data** — the Worker enforces namespace isolation. Your GitHub username is your storage prefix. You cannot read, list, or delete another user's checkpoints.274 275### What we DO NOT do276- We do NOT sell, share, or analyze your data.277- We do NOT train models on your checkpoints.278- We do NOT log your GitHub token value.279- We do NOT track your usage beyond standard Cloudflare request metrics.280 281### What we disclose282- The checkpoint API Worker code is in our private repo. We plan to open-source it.283- Cloudflare's privacy policy applies to request metadata: https://www.cloudflare.com/privacypolicy/284- Checkpoints are automatically deleted after 7 days. We do not keep backups.285 286### If you don't trust us287That's fair. You can:2881. **Use just the HF dataset** — no account, no tokens, no interaction with our API. Just `huggingface-cli download`.2892. **Save checkpoints locally** — skip the API entirely. Save to `/workspace/` and accept the risk of losing work on preemption.2903. **Inspect the Worker** — we'll open-source it. Until then, the API surface is 5 endpoints, ~150 lines of JavaScript, zero dependencies.291 292---293 294## Provenance295 296- **Source:** [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) (CommonCrawl-derived, by Hugging Face). Our docs come from the `willdepueoai/parameter-golf` competition mirror.297- **SP1024 tokenization:** SentencePiece BPE, 1024 tokens — from the [openai/parameter-golf](https://github.com/openai/parameter-golf) competition repo, `data/tokenizer_specs.json`.298- **SP4096 / SP8192 / SP12288 / SP16384 tokenizations:** SentencePiece BPE models trained by LightSpeedUp on 1 M FineWeb docs decoded byte-exactly from the canonical SP1024 shards (SP1024 uses `byte_fallback=True`, so the decode is lossless). Trainer settings mirror the competition's `build_sentencepiece_tokenizer`: `character_coverage=0.999`, `byte_fallback=True`, `split_digits=True`, `normalization_rule_name=nmt_nfkc`, `add_dummy_prefix=False`, `hard_vocab_limit=False`.299- **byte260 tokenization:** direct UTF-8 byte mapping. Vocab 260 = 256 bytes + 4 reserved specials. Bytes `0x00..0xFF` map to ids `0..255` directly (no offset). Reserved specials: `pad_id=256, bos_id=257, eos_id=258, unk_id=259`. Encoding is `[257] + list(text.encode("utf-8"))`; `append_eos=False`. No SentencePiece model involved. Byte-accounting LUT is trivial: `base_bytes_lut[tok] = 1 if tok < 256 else 0`. **Note on convention:** this differs from the `openai/parameter-golf` `PureByteTokenizer` defaults (which put specials at 0..3 and offset bytes to 4..259). When loading these shards into the canonical `train_gpt.py --variant byte260` path, use a loader that matches the convention above.300- **Scylla v2 tokenization:** corrected byte-exact TokenMonster vocab (1254 tokens, logical 1178, `bos_id=1253`) from [@simon-marcus](https://github.com/simon-marcus)'s [PR #1314](https://github.com/openai/parameter-golf/pull/1314). Regime: `charset=none`, `capcode=0`, `normalization=none`, explicit full byte fallback, latin-1 byte interpretation, synthetic zero-byte BOS per doc.301- **Scylla v1 tokenization (deprecated):** original 998-token TokenMonster vocab from [@simon-marcus](https://github.com/simon-marcus)'s [PR #1143](https://github.com/openai/parameter-golf/pull/1143). Superseded by v2 due to the byte-accounting issue in [Issue #897](https://github.com/openai/parameter-golf/issues/897).302- **No modification** to token sequences on any variant — byte-identical to what you'd produce by running the same pipeline yourself.303 304### Attribution Chain305 306CommonCrawl (CC-BY) → Hugging Face FineWeb (ODC-By 1.0) → willdepueoai/parameter-golf (ODC-By 1.0) → This dataset (ODC-By 1.0)307 308## License309 310**Data:** [Open Data Commons Attribution License (ODC-By 1.0)](https://opendatacommons.org/licenses/by/1-0/)311 312**Code & Tools:** Apache 2.0 — see [PATENTS.md](PATENTS.md)313 314---315 316## Related Community Resources317 318- **[`sproos/parameter-golf-tokenizers`](https://huggingface.co/sproos/parameter-golf-tokenizers)** — a complementary community mirror that publishes `fineweb10B_{sp1024, sp2048, sp4096, sp8192}/` pre-tokenized shards plus the corresponding SentencePiece `.model` / `.vocab` files. If you only need the SP variants in the 1K–8K vocab range, sproos is a direct source. Our `LightSpeedUp/parameter-golf-data` drop is complementary: we add the larger SP variants (12288, 16384), the corrected Scylla v2, and byte260, and we bundle the free checkpoint-persistence API + Docker image for end-to-end training on free-tier GPUs.319 320## Roadmap321 322- Cloudflare R2 mirror for HF-rate-limited users (coming soon)323- Automated checkpoint save/resume in the boot script324- Open-source the CF Worker code325 326## Community327 328- [The Agora](https://matotezitanka.github.io/parameter-golf) — live leaderboard + compliance tracker329- [Issue #942](https://github.com/openai/parameter-golf/issues/942) — compute resources discussion330- [Issue #140](https://github.com/openai/parameter-golf/issues/140) — competition discussion thread331