Ayushnangia/dolma3-hq-2M-modernbert

Dolma3 High-Quality 2M (ModernBERT Filtered) A curated subset of 2 million high-quality text samples from allenai/dolma3_dolmino_mix-100B-1125, filtered to fit within ModernBERT's 8192 token context window. Dataset Description This dataset is designed for pretraining diffusion language models based on ModernBERT. Each sample has been: Source filtered: Only from ingredient1-common_crawl-high-quality folders (highest quality web text) Length filtered: Minimum 200… See the full description on the dataset page: https://huggingface.co/datasets/Ayushnangia/dolma3-hq-2M-modernbert.