ncylich/temporal-moe-corpus

temporal-moe-corpus

Tokenized training corpus for the Temporal-MoE experiments, 31.3 GiB.

This repository holds the tokenized corpus in Megatron indexed-dataset format, plus the 16k tokenizer. It is what the training runs actually read. The raw web-crawl text is not hosted here, because it is a byte-reproducible subset of a public dataset. The exact recipe and the checksums needed to verify a reproduction are below.

Contents

dclm_tokenized/    22 x part<NN>_text_document.{bin,idx}   tokenized with EleutherAI/pythia-12b (50k vocab)
tok16k_full/       11 x part<NN>_text_document.{bin,idx}   tokenized with the 16k tokenizer below
tokenizer/         tokenizer.json, tokenizer_config.json    16k byte-level BPE, trained on this corpus
parquet_sha256.txt  sha256 of each of the 88 upstream parquet shards
jsonl_sha256.txt    sha256 of each of the 22 intermediate part<NN>.jsonl files

.bin files are uint16 token ids. .idx files are the Megatron index. Load them with megatron.core.datasets.indexed_dataset, or point --data-path at the part<NN>_text_document prefix.

Reproducing the raw input, exactly

Step 1, upstream source

Everything derives from one pinned dataset revision:

parquet_sha256.txt in this repository lists the sha256 of every one of those 88 shards as they were downloaded. Verify against it before proceeding.

Step 2, parquet to JSONL

experiments/data/download_parts.py in the code repository, run with 4 shards per part, produces 22 files part00.jsonl through part21.jsonl, each a {"text": ...} object per line with empty documents dropped.

This step is deterministic. Shard indices are derived from the part index, ThreadPoolExecutor.map yields results in input order rather than completion order, and json.dumps on a single-key dict is stable. Re-running it against the pinned revision reproduces the same bytes.

jsonl_sha256.txt lists the sha256 of all 22 intermediate files, so a reproduction can be verified bit for bit without this repository hosting them.

Step 3, JSONL to tokenized shards

experiments/data/fast_tokenize.py in the code repository, with EOD=0 and add_special_tokens=False, writing uint16 via IndexedDatasetBuilder. Each part is tokenized independently, so there is no cross-part ordering dependency.

This step was verified empirically, not merely argued. Re-tokenizing the first 2000 documents of a part and comparing against the stored .bin gives a byte-identical result:

The 16k tokenizer

tokenizer/ is a 16k-vocab byte-level BPE trained by experiments/data/train_tok16k.py on a text sample drawn from part00.jsonl and part01.jsonl, vocab size 16000, min_frequency=2, single special token <|endoftext|> with id 0.

The trained tokenizer is shipped here directly, so reproducing it is not required in order to use or re-derive the corpus. It is the artifact, not an intermediate.

Why the raw text is not hosted

The 22 dclm_parts JSONL files and the tokenizer training sample are unfiltered DCLM web crawl. A scan of that text found third-party material that is not ours to redistribute, including private key blocks, cloud access key ids, and roughly 104,000 lines containing email addresses. That content is already public as part of DCLM, and the pinned revision plus the checksums above let anyone reconstruct the exact bytes, so nothing about reproducibility is lost by not mirroring it here.

Treat any credential encountered in reconstructed DCLM text as compromised and unusable.

MANIFEST.csv and the cited column

A manifest covering every file in all four repositories lives in the code repository. It has seven columns: local_path, hf_repo, hf_path, bytes, sha256, run_name, cited.

The cited column marks whether a run is referenced by a results table in results/ablations/*.csv or by the paper:

• —cited, the run backs a published number. There are 58 of these.
• —uncited, the run is infrastructure validation, a smoke test, a throughput probe, or an aborted run. It is kept for completeness and reproducibility, not because a table depends on it. There are 13 of these.
• —empty, the file is not scoped to a single run, for example a batch log or an evaluation output.

Every sha256 in the manifest was computed on this disk before upload and each file was verified to exist remotely with a matching byte size.

Links

• —Code: <https://github.com/ncylich/temporal-moe>
• —Paper: Temporal-MoE (short paper), see the paper/ directory in the code repository
• —Upstream platform this work forks: FLAME-MoE, arXiv:2505.20225

Companion repositories

• —`ncylich/temporal-moe-ckpts`, Megatron training checkpoints
• —`ncylich/temporal-moe-router-adapt`, router adaptation safetensors
• —`ncylich/temporal-moe-extras`, captures, merged model, result tables, figures
• —`ncylich/temporal-moe-corpus`, tokenized training corpus

Provenance

Trained with a personal fork of FLAME-MoE. Not affiliated with or endorsed by the FLAME-MoE authors.