ssuresh/nemo-stage1-50M-samples

NeMo Stage1 Pretraining Dataset - 50M Samples

This dataset contains 50 million text samples for NeMo model pretraining (Stage 1). The dataset is organized in chunks for efficient loading and processing.

Dataset Details

• —Total Samples: ~50,000,000
• —Format: JSONL (JSON Lines)
• —Structure: Each sample contains {"id": number, "text": "content"}
• —Chunks: 47 files (chunk000.jsonl to chunk046.jsonl)
• —Samples per chunk: ~1,000,000
• —Language: English
• —Task: Text generation pretraining

Usage

Load the entire dataset:

from datasets import load_dataset

# Load all chunks
dataset = load_dataset('ssuresh/nemo-stage1-50M-samples', split='train')
print(f"Loaded {len(dataset)} samples")

Load specific chunks:

# Load a specific chunk
chunk_dataset = load_dataset('ssuresh/nemo-stage1-50M-samples', 
                           data_files='chunks/chunk_000.jsonl', 
                           split='train')
print(f"Loaded {len(chunk_dataset)} samples from chunk_000")

Load multiple chunks:

# Load multiple chunks
chunk_files = [f'chunks/chunk_{i:03d}.jsonl' for i in range(5)]  # First 5 chunks
multi_chunk_dataset = load_dataset('ssuresh/nemo-stage1-50M-samples', 
                                 data_files=chunk_files, 
                                 split='train')
print(f"Loaded {len(multi_chunk_dataset)} samples from 5 chunks")

Dataset Structure

Each sample in the dataset has the following structure:

{
  "id": 0,
  "text": "Your text content here..."
}

Chunk Information

• —chunk_000.jsonl to chunk_046.jsonl: Individual chunk files
• —Each chunk contains approximately 1,000,000 samples
• —Chunks are numbered sequentially from 000 to 046
• —Total: 47 chunks

Sample Data

The first 100 samples are available in sample_data.json for inspection and understanding the data format.

Memory-Efficient Loading

This dataset is designed for memory-efficient loading:

• —Load individual chunks to avoid memory issues
• —Use streaming for very large datasets
• —Chunk-based processing for distributed training

Training Usage

For NeMo training, you can use this dataset with the use_processed_datasets: true setting in your configuration:

data:
  use_processed_datasets: true
  dataset_name: "ssuresh/nemo-stage1-50M-samples"

License

This dataset is released under the Apache 2.0 License.

Citation

If you use this dataset in your research, please cite:

@dataset{nemo_stage1_50m,
  title={NeMo Stage1 Pretraining Dataset - 50M Samples},
  author={Suresh},
  year={2024},
  url={https://huggingface.co/datasets/ssuresh/nemo-stage1-50M-samples}
}

Contact

For questions or issues with this dataset, please open an issue on the Hugging Face Hub or contact the dataset maintainer.