shenjiarun/qwen3-c4-megatron

qwen3-c4-megatron Megatron-LM indexed dataset files for Qwen3 tokenizer preprocessing on the C4 corpus. Directory layout qwen3-c4-megatron/ ├── README.md ├── metadata.json ├── manifest.sha256 └── data/ ├── c4_full.bin └── c4_full.idx Files data/c4_full.bin: Megatron-LM binary token data. data/c4_full.idx: Megatron-LM index file for c4_full.bin. metadata.json: Dataset metadata and preprocessing configuration. manifest.sha256: SHA256… See the full description on the dataset page: https://huggingface.co/datasets/shenjiarun/qwen3-c4-megatron.