modernbert
midtraining_mix_modernbert_filtered_documentsmailroom-modernbert-training
mailroom-modernbert-training
Cleaned + prepared hierarchical-classification training set for the
ModernBERT-base ingest fast-path — the fine-tuning surface of the
mailroom-ml synthetic-data layer.
The layer (end to end)
Lucius-Morningstar/mailroom-dataset corpus (GT labels, canonical v9)
│ (working copy, pinned)
▼
THIS REPO (mailroom-modernbert-training) @ pinned revision
│ training/train_modernbert.py (--data <repo>)
▼… See the full description on the dataset page: https://huggingface.co/datasets/Lucius-Morningstar/mailroom-modernbert-training.ni-ood-dataset-20250131-modernbert-train-kmeans-dim128-20250312dolma3-hq-2M-modernbert
Dolma3 High-Quality 2M (ModernBERT Filtered)
A curated subset of 2 million high-quality text samples from allenai/dolma3_dolmino_mix-100B-1125, filtered to fit within ModernBERT's 8192 token context window.
Dataset Description
This dataset is designed for pretraining diffusion language models based on ModernBERT. Each sample has been:
Source filtered: Only from ingredient1-common_crawl-high-quality folders (highest quality web text)
Length filtered: Minimum 200… See the full description on the dataset page: https://huggingface.co/datasets/Ayushnangia/dolma3-hq-2M-modernbert.wikitext-tags-modernbertdata_ablation_full59K-modernbert-split-kmeans-dim768-20250218