sfanm/d24-midtrain-olmo3-5b

d24 Midtrain — OLMo-3 Dolmino (5B, chunked)

A 5.00B-token reproduction of OLMo 3's Dolma-3 Dolmino mid-train mix, built by taking a fraction of each component of `allenai/dolma3_dolmino_mix-100B-1025`. Unlike the smaller `d24-midtrain-olmo3` (which dropped documents over 2048 tokens — silently zeroing the long reasoning-trace and PDF components), this build chunks long documents into 2048-token windows (decoded back to text), so every component keeps ~100% of its tokens and reaches its target — the realized mix matches OLMo-3's true proportions (see the OLMo-3 target % column ≈ the kept share).

• —11,066,837 documents (post-chunk), ~5.00B GPT-2 tokens of plain text.
• —Each row: component (str) and text (str), every row ≤2048 tokens.
• —Chunking caveat: long chain-of-thought traces and PDFs are split at 2048-token boundaries — each window is its own row (its own EOS at tokenization), so reasoning chains are present but fragmented across rows.

Components (realized ≈ OLMo-3 target)

Load

from datasets import load_dataset
ds = load_dataset("sfanm/d24-midtrain-olmo3-5b", split="train")
ds = ds.filter(lambda r: r["component"] == "qwq-reasoning-traces")   # now non-empty (was ~0 in the dropped build)

Relationship to the 2.3B build

• —`d24-midtrain-olmo3` — ~2.1B, long docs dropped → web/short-QA heavy, ~0 reasoning traces (the corpus the `d24-*-olmo3-2.3B` models trained on).
• —this — ~5B, long docs chunked → faithful OLMo-3 proportions incl. the reasoning traces.

License / attribution

Derived from `allenai/dolma3_dolmino_mix-100B-1025` (itself a mix of many sources — see its card). Provided as-is for research; refer to the upstream licenses.