Datasetpublic

sfanm/d24-midtrain-olmo3

d24 Midtrain — OLMo-3 Dolmino subsample A length-filtered (≤2048 GPT-2 tokens) subsample of OLMo 3's Dolma-3 Dolmino mid-train mix, used for the d24 v1base-olmo3 midtrain. Built by taking a uniform fraction of each component of allenai/dolma3_dolmino_mix-100B-1025 (so the mix proportions are preserved), then dropping documents >2048 tokens — which naturally shrinks the long-doc components (reasoning traces, olmOCR PDFs) that don't fit the 2048 context. 5,902,548 documents… See the full description on the dataset page: https://huggingface.co/datasets/sfanm/d24-midtrain-olmo3.

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes164downloads

Dataset Card

d24 Midtrain — OLMo-3 Dolmino subsample

A length-filtered (≤2048 GPT-2 tokens) subsample of OLMo 3's Dolma-3 Dolmino mid-train mix, used for the d24 v1base-olmo3 midtrain. Built by taking a uniform fraction of each component of `allenai/dolma3_dolmino_mix-100B-1025` (so the mix proportions are preserved), then dropping documents >2048 tokens — which naturally shrinks the long-doc components (reasoning traces, olmOCR PDFs) that don't fit the 2048 context.

—5,902,548 documents, ~2109M GPT-2 tokens of plain text.
—Each row: component (str) and text (str).
—The OLMo-3 target % column is the upstream mix proportion (pre length-filter); the kept document share differs because the filter removes long docs unevenly.

Components

component	documents	~tokens	OLMo-3 target %
`common_crawl-high-quality`	672,640	~598M	22.50%
`dolmino-math`	714,086	~266M	10.70%
`cranecode`	238,782	~169M	10.00%
`stack_edu-fim`	388,598	~164M	10.00%
`reddit_to_flashcards`	2,190,171	~148M	5.90%
`cranemath`	269,960	~135M	5.63%
`dolmino_1-flan`	330,626	~130M	5.00%
`nemotron-synth-qa`	225,305	~125M	5.00%
`olmocr_science_pdfs`	12	~0M	5.00%
`stem-heavy-crawl`	243,244	~131M	5.00%
`wiki_to_rcqa`	342,484	~75M	3.00%
`general_reasoning_mix`	38,505	~41M	1.87%
`qwq-reasoning-traces`	3	~0M	1.87%
`megamatt`	87,014	~42M	1.73%
`llama_nemotron-reasoning-traces`	4	~0M	1.25%
`openthoughts2-reasoning-traces`	2,853	~3M	1.25%
`tulu-3-sft`	66,517	~29M	1.10%
`tinymath-mind`	49,788	~22M	0.90%
`omr-rewrite-fullthoughts`	6,057	~6M	0.85%
`code-meta-reasoning`	6,929	~13M	0.46%
`math-meta-reasoning`	4,503	~5M	0.38%
`gemini-reasoning-traces`	323	~0M	0.25%
`tinymath-pot`	20,730	~6M	0.24%
`program_verifiable`	3,414	~2M	0.16%

Load

python

from datasets import load_dataset
ds = load_dataset("sfanm/d24-midtrain-olmo3", split="train")
ds = ds.filter(lambda r: r["component"] == "dolmino-math")

Finding (see the model cards / research log)

For the narrow GSM8K target, this broad/clean OLMo-3 mix (~25% math) lost to the math-heavy pool (~70% math): the resulting SFT had lower train loss (format familiarity from the instruction/QA components) but worse val and GSM8K — memorization, not capability. Kept public for reproducibility of that result.

Model trained on this (midtrain → SFT)

—`sfanm/d24-midtrain-v1base-olmo3-2.3B` → `sfanm/d24-sft-v1base-olmo3-2.3B`

License / attribution

Derived from `allenai/dolma3_dolmino_mix-100B-1025` (itself a mix of many sources — see its card). Provided as-is for research; refer to the upstream licenses.