dlab-spp/corpus-verification

SPP Corpus Verification

Checksums and document-boundary indices for verifying a rebuilt copy of the Synthetic Persona Pretraining (SPP) training corpus, byte for byte.

The Megatron token streams themselves are 2.17 TB (annotated.bin 421 GB, compact.bin 1.75 TB) and are fully derived from the published reflections, the uid manifest, and the tokenizer recipe — so they are not published. These .idx sidecars carry per-document boundaries and lengths, which is enough to prove an independently rebuilt .bin matches ours at ~200× less data.

🔁 Manifest (what to rebuild from): `dlab-spp/corpus-1T-manifest` 📄 Reflections: `dlab-spp/reflection-50m`

Files

How to use

Rebuild the streams per REPRODUCTION.md §6, then compare your .idx document boundaries against these.

The failure this is most likely to catch: using transformers.AutoTokenizer instead of the Rust tokenizers library. The two disagree on \n\n — Rust emits a single token 1116, AutoTokenizer emits [198, 198] — so the wrong tokenizer shifts boundaries on essentially every document containing a blank line.

Two other things worth asserting while you are here: the truncation cap is 1919 content tokens (enable_truncation(max_length=1920), EOS counts toward the limit), with 25.4% of annotated rows sitting exactly at it; and the measured annotated-stream total is 107,007,683,660 tokens, about 3% below the 110.30B estimate in the subsample metadata, which was computed on untruncated text.

License and attribution

Released under the Open Data Commons Attribution License (ODC-BY 1.0), inherited from the upstream source.

Contains information from `allenai/dolma3_mix-6T`, made available under the Open Data Commons Attribution License (ODC-BY 1.0).

Please cite Olmo 3 (arXiv:2512.13961) and observe AI2's Responsible Use Guidelines.

Citation

@misc{minder2026syntheticpersonapretrainingalignment,
      title={Synthetic Persona Pretraining: Alignment from Token Zero},
      author={Julian Minder and Viktor Moskvoretskii and Raghav Singhal and Difan Jiao and Andy Arditi and Shaobo Cui and Yiderigun Borjigin and Kartik Bali and Stefan Krsteski and Harsh Raj and Huu Nguyen and Jannik Brinkmann and Ashton Anderson and Roland Aydin and Robert West},
      year={2026},
      eprint={2608.13482},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.13482},
}