open-athena/snowball-replay-index

Snowball replay index

This dataset is a compact membership and ordering index for an approximate replay of Snowball's 10,372,343,704,053-token data store. It contains no source text or token arrays. The 6,301 Parquet files contain three columns:

• —source_id: logical source key; join it to the source_id field in sources.json
• —document_id: the retained XXH3-128 content hash as 16 bytes
• —bucket_id: domain_cluster * 5 + quality_bucket

Document join contract

document_id is not the upstream row ID. Fetch the public input named in sources.json, apply its pinned Marin transform and normalize_config, and hash the resulting text UTF-8 bytes with unseeded XXH3-128 using the algorithm's standard default secret. Marin formatted the unsigned integer digest as a zero-padded 32-character lowercase hexadecimal value; the Parquet value is bytes.fromhex(hex_id). Convert a stored value back with document_id.hex(). The historical normalizer preserved an upstream ID separately as source_id, but that upstream ID is not included here. Matching therefore requires normalized content.

Clone Marin at 871f36b38184e659e471fa6d0e8e660f6170c177 to use the historical transform and normalization implementation. For transformed sources, transform_module and replay_entrypoint name the code path. For generic Hugging Face sources, an empty file_patterns list means every repository file at the pinned revision that passes relative_input_path and file_extensions in normalize_config; there is no datasets-library split selection. Sources backed by changing URLs are marked approximate.

Store order and buckets

Read files in numeric task_id order. Preserve row order within each file. To rebuild a bucket stream, filter each file by bucket_id without reordering and concatenate the results. Domain clusters are 0 through 39. Quality bucket 0 has score below 0.2; bucket 1 is [0.2, 0.4); bucket 2 is [0.4, 0.6); bucket 3 is [0.6, 0.8); and bucket 4 is 0.8 or higher. bucket_id = domain_cluster * 5 + quality_bucket, yielding values 0 through 199. The tail sampler component is the ordered concatenation of the bucket IDs listed in mix.json.

After joining, tokenize normalized text with the pinned marin-community/marin-tokenizer revision, enforce the EOS policy in mix.json, and concatenate each bucket's documents in index order.

Mixture replay

manifest.json records counts and SHA-256 hashes. sources.json maps all 114 logical sources to public repositories or download URLs and states known fidelity limits. mix.json records the two-phase sampler configuration recovered from the production W&B run.

After downloading this dataset, run python validate.py --repo-id open-athena/snowball-replay-index to check public visibility, metadata totals, the exact repository file set, every selection-shard size, sampled Parquet SHA-256 hashes, and anonymous upstream revisions.

mix.json records the configured two-phase mixture, batch schedule, random-key derivation, and implementation pins. The phase change starts at training update 127,500. Levanter converts that update boundary to a global sequence offset using the batch schedule, rounds component counts down within each 49,152-sequence mixture block, and assigns the remainder to the largest component. Finite components restart modulo their length. Dataset shuffles and mixture blocks use separate keys derived from the trainer's data key. Reproducing those details requires the pinned Levanter implementation, not only the weights.

This is an approximate campaign replay, not a claim of exact observed token exposure. The production campaign resumed across a 4,096-batch run, an 8,192-batch run, and a 65,536-sequence cooldown. Its W&B counters and known checkpoint boundaries are recorded, but exact continuation offsets require the original checkpoint sampler state or independent verification. Exact source reconstruction also depends on continued upstream availability and deterministic historical transforms. HPLT and NSF did not have immutable upstream snapshots. Some Hugging Face repositories require authentication or acceptance of their own terms.

The transformation code is in marin-community/marin. Store construction semantics are pinned to commit 871f36b38184e659e471fa6d0e8e660f6170c177. The Snowball mixture launcher is pinned to commit 8b1d3a485d0432e08953b90b0e402b411eed4261.