sfanm/d12-pretrain-climbmix15B-dolmino5B

D12 — 124M pretrain on ClimbMix + Dolmino (20B tokens)

The loadable model at this repository's root is the terminal checkpoint of a 124M-parameter decoder trained from scratch on a 20B-token mixture: 75% ClimbMix web-scale pretraining text and 25% of the OLMo-3 Dolmino mid-training mix, interleaved uniformly through the whole run. The run consumed exactly 20,000,014,336 training tokens (38,147 optimizer iterations at 256 × 2,048), of which 15.0B came from ClimbMix and 5.0B from Dolmino.

Architecture

The Transformers class is LlamaForCausalLM, but this is a from-scratch model with the GPT-2 tokenizer (the same family as the sfanm/d24-* models, at half the depth and hidden size), not a Llama-family pretrained checkpoint.

Data

Both corpora were tokenized with the GPT-2 tokenizer and sampled at the token level, not physically resized: every training batch is about 75% ClimbMix and 25% Dolmino. Every document ends with exactly one <|endoftext|> token (id 50256, the model's eos_token_id); no beginning-of-sequence token was prepended, which matches the tokenizer's default behaviour. Token ids 50257–50303 exist only as vocabulary padding and never occur in the data.

• —ClimbMix: a 100B-token tokenization of karpathy/climbmix-400b-shuffle, 62 training shards; 15.0B tokens were drawn, at most 15.3% of any shard.
• —Dolmino: allenai/dolma3_dolmino_mix-100B-1025 (OLMo-3's mid-training mix at its released component proportions), 62 training shards; 5.0B tokens were drawn, at most 4.5% of any shard.

No document was seen twice. One held-out shard per corpus served as the validation and test split, mixed with the same 75/25 weights.

Training

Sequence length 2,048, global batch 256, AdamW (β 0.9/0.95, ε 1e-8) at peak LR 6e-4, weight decay 0.1, gradient clip 1.0. The schedule is WSD: 200 warmup steps, constant 6e-4 through iteration 30,518, then a 7,629-step cosine cooldown to 6e-5. Final held-out language-model loss on the mixed validation split was 2.5535 (perplexity 12.9); on the test split 2.5690 (perplexity 13.1). Training loss fell from 2.73 at the end of the constant phase to 2.64 at the end of the cooldown.

The run was trained with Megatron-Bridge (NeMo 26.04 container) on ALCF Polaris, on 8 nodes × 4 A100-40GB, in about 36 node-hours.

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sfanm/d12-pretrain-climbmix15B-dolmino5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

This is a base language model and has no chat template. The resumable Megatron distributed checkpoint of the terminal iteration is published under megatron/iter_0038147 (model, optimizer and scheduler state).

This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.