Ayushnangia/dolma3-hq-2M-modernbert

Dolma3 High-Quality 2M (ModernBERT Filtered)

A curated subset of 2 million high-quality text samples from allenai/dolma3_dolmino_mix-100B-1125, filtered to fit within ModernBERT's 8192 token context window.

Dataset Description

This dataset is designed for pretraining diffusion language models based on ModernBERT. Each sample has been:

1. 1.Source filtered: Only from ingredient1-common_crawl-high-quality folders (highest quality web text)
2. 2.Length filtered: Minimum 200 characters
3. 3.Token filtered: Maximum 8192 tokens using ModernBERT tokenizer (samples exceeding this are excluded, not truncated)
4. 4.Randomly sampled: True random sampling from 2.4M collected samples down to 2M

Usage

from datasets import load_dataset

dataset = load_dataset("Ayushnangia/dolma3-hq-2M-modernbert")
print(f"Samples: {len(dataset['train']):,}")
print(dataset['train'][0]['text'][:500])

For ModernBERT Diffusion LM Pretraining

python scripts/mb_pretrain.py \
    --dataset Ayushnangia/dolma3-hq-2M-modernbert \
    --text-column text

Dataset Statistics

Source

• —Original dataset: allenai/dolma3_dolmino_mix-100B-1125
• —Tokenizer: answerdotai/ModernBERT-large

License

Apache 2.0 (following Dolma3's license)

Citation

If you use this dataset, please cite the original Dolma3 dataset:

@article{dolma,
  title={Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research},
  author={Soldaini, Luca and others},
  journal={arXiv preprint},
  year={2024}
}