Mr-Philo/dolma3_dolmino_megatron_tokenize
Dolma 3 / Dolmino Megatron-LM indexed dataset This repository contains immutable Megatron-LM indexed datasets (.bin and .idx) produced from pinned Dolma 3 and Dolmino releases. It intentionally contains no training checkpoints, experiment outputs, logs, or dataset caches. The indexed payloads were derived from these pinned public datasets: allenai/dolma3_mix-150B-1025@afa92bfb22366821c5e6cd427cdd036b34b713ef… See the full description on the dataset page: https://huggingface.co/datasets/Mr-Philo/dolma3_dolmino_megatron_tokenize.
Dolma 3 / Dolmino Megatron-LM indexed dataset
This repository contains immutable Megatron-LM indexed datasets (.bin and .idx) produced from pinned Dolma 3 and Dolmino releases. It intentionally contains no training checkpoints, experiment outputs, logs, or dataset caches.
The indexed payloads were derived from these pinned public datasets:
allenai/dolma3_mix-150B-1025@afa92bfb22366821c5e6cd427cdd036b34b713efallenai/dolma3_dolmino_mix-100B-1125@f23aa129fda8335ba9760057bcc1f0c02f3d068b
Both source repository cards declare the ODC-By license. The .bin files are token IDs, not decoded source text. Each .idx file is the matching Megatron-LM document index. The directory name records whether the GLA/Llama 32K tokenizer or the Qwen3.5 tokenizer was used.
Inventory: 64 files, 2.1 TiB. See manifest.json for exact repository paths and sizes.