Datasetpublic

shenjiarun/qwen3-c4-megatron

qwen3-c4-megatron Megatron-LM indexed dataset files for Qwen3 tokenizer preprocessing on the C4 corpus. Directory layout qwen3-c4-megatron/ ├── README.md ├── metadata.json ├── manifest.sha256 └── data/ ├── c4_full.bin └── c4_full.idx Files data/c4_full.bin: Megatron-LM binary token data. data/c4_full.idx: Megatron-LM index file for c4_full.bin. metadata.json: Dataset metadata and preprocessing configuration. manifest.sha256: SHA256… See the full description on the dataset page: https://huggingface.co/datasets/shenjiarun/qwen3-c4-megatron.

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes78downloads

Dataset Card

qwen3-c4-megatron

Megatron-LM indexed dataset files for Qwen3 tokenizer preprocessing on the C4 corpus.

Directory layout

text

qwen3-c4-megatron/
├── README.md
├── metadata.json
├── manifest.sha256
└── data/
    ├── c4_full.bin
    └── c4_full.idx

Files

—data/c4_full.bin: Megatron-LM binary token data.
—data/c4_full.idx: Megatron-LM index file for c4_full.bin.
—metadata.json: Dataset metadata and preprocessing configuration.
—manifest.sha256: SHA256 checksums for all files except the manifest itself.

Usage

Use the dataset prefix without the file extension when passing it to Megatron-LM:

bash

--data-path /225045001/Megatron-LM/qwen3-c4-megatron/data/c4_full