gaotang/loopmoe-fineweb-edu-100bt-mcore

LoopMoE FineWeb-Edu 100BT (Megatron indexed)

Release status: `complete`

This is the exact pretokenized FineWeb-Edu 100BT corpus prepared for the reviewed LoopMoE M1 Dense/Loop/Dual and Fast-Slow experiment contracts. This release does not define the M2--M5 train schedules. It contains 64 training indexed-dataset shards and one fixed validation shard. Each prefix has a Megatron .bin/.idx pair and a sanitized .stats.json record. There is no separate test split.

The validation source is exactly sample/100BT/000_00000.parquet and is excluded from training. manifest.json records a full SHA-256 for all 195 artifacts. order_contract.json records the exact C-sorted prefix order, weights, seed, sequence length, batch size, train horizon, and evaluation horizon. The provenance/ directory contains sanitized source-download, preprocessing, verification, cache-preparation, and exact launcher references.

After downloading the repository, this standalone audit rehashes every artifact (the verifier refuses an uploading/incomplete release):

hf download gaotang/loopmoe-fineweb-edu-100bt-mcore --repo-type dataset --local-dir "$DATASET_ROOT"
python provenance/verify_download.py "$DATASET_ROOT" --workers 4

release_status.json reports both the count/bytes for all 195 manifest artifacts (.bin, .idx, and .stats.json) and the separate count/bytes for the 130 indexed-data files (.bin and .idx).

Each source document ends with EOD. Loss is not masked at EOD, position IDs and attention masks are not reset at document boundaries, Megatron's extra next-token label is enabled, and the dataloader does not materialize an attention mask.

The M1 contract is seed 1234, sequence length 4096, GBS 128, 67,083 updates (35,170,811,904 tokens), and one final-only validation pass. Periodic validation is suppressed with --eval-interval 67084; the final pass uses --eval-iters 95 and --eval-global-batch-size 128 (12,160 sequences / 49,807,360 tokens).

With DP=8, zero-based optimizer update i consumes shuffled global sequence indices [128*i, 128*(i+1)); changing micro-batch size only partitions those same indices across microsteps and ranks. Dense Standard, LoopFinal, Dual, and Fast-Slow therefore share the data order. Evaluation consumes the first 12,160 shuffled validation entries. Path-bound Megatron cache files are deliberately omitted: their descriptions and MD5 names encode absolute dataset/tokenizer paths. Rebuild the exact M1 cache in a new empty cache directory from the source snapshot identified by provenance/provenance_manifest.json:

python provenance/prepare_m1_cache.py   --megatron-root /path/to/muploco   --dataset-root "$DATASET_ROOT"   --tokenizer-model /path/to/original/tokenizer.model   --data-cache-path /new/empty/cache/path   --execute

Never launch concurrent cache builders against the same cache path. The cache helper invokes the same full download verifier before it renders or executes a command, so a separate standalone verification immediately before cache preparation is unnecessary. The exact source is also linked from https://github.com/GaotangLi/LoopAug; the recorded commit and SHA-256 bindings, rather than a moving branch, are authoritative.

The tokenizer binary is deliberately not redistributed. Obtain original/tokenizer.model from the gated meta-llama/Meta-Llama-3-8B repository and verify SHA-256 82e9d31979e92ab929cd544440f129d9ecd797b69e327f80f17e1c50d5551b55. The indexed data uses vocabulary size 128,256, EOD id 128,001, and 256 special tokens. Meta's applicable model access/license terms govern the tokenizer; this dataset's ODC-By declaration does not grant rights to that separate binary.

Source: `HuggingFaceFW/fineweb-edu`, pinned at 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9. FineWeb-Edu is distributed under ODC-By; users must preserve its attribution and comply with the source terms.

While the status is uploading/incomplete, do not consume the repository. The status changes to complete only after every expected remote path and size, plus every SHA-256 exposed by Hub LFS metadata, matches the release manifest.