sfanm/d12-pretrain-climbmix30B-dolmino10B

D12 v2 — 124M pretrain on ClimbMix + Dolmino (40B tokens)

The loadable model at this repository's root is the terminal checkpoint of a 124M-parameter decoder trained from scratch on a 40B-token mixture: 75% ClimbMix web-scale pretraining text and 25% of the OLMo-3 Dolmino mid-training mix, interleaved uniformly through the whole run. The run consumed exactly 40,000,028,672 training tokens (76,294 optimizer iterations at 256 × 2,048), of which 30.0B came from ClimbMix and 10.0B from Dolmino. It is the same recipe as sfanm/d12-pretrain-climbmix15B-dolmino5B with twice the data; that 20B-token model is the natural comparison point.

Architecture

The Transformers class is LlamaForCausalLM, but this is a from-scratch model with the GPT-2 tokenizer (the same family as the sfanm/d24-* models, at half the depth and hidden size), not a Llama-family pretrained checkpoint.

Data

Both corpora were tokenized with the GPT-2 tokenizer and sampled at the token level, not physically resized: every training batch is about 75% ClimbMix and 25% Dolmino. Every document ends with exactly one <|endoftext|> token (id 50256, the model's eos_token_id); no beginning-of-sequence token was prepended, which matches the tokenizer's default behaviour. Token ids 50257–50303 exist only as vocabulary padding and never occur in the data.

• —ClimbMix: a 100B-token tokenization of karpathy/climbmix-400b-shuffle, 62 training shards; 30.0B tokens were drawn, at most 30.6% of any shard.
• —Dolmino: allenai/dolma3_dolmino_mix-100B-1025 (OLMo-3's mid-training mix at its released component proportions), 62 training shards; 10.0B tokens were drawn, at most 8.9% of any shard.

No document was seen twice. One held-out shard per corpus served as the validation and test split, mixed with the same 75/25 weights.

Training

Sequence length 2,048, global batch 256, AdamW (β 0.9/0.95, ε 1e-8) at peak LR 6e-4, weight decay 0.1, gradient clip 1.0. The schedule is WSD: 200 warmup steps, constant 6e-4 through iteration 61,035, then a 15,259-step cosine cooldown to 6e-5. Final held-out language-model loss on the mixed validation split was 2.4573 (perplexity 11.7); on the test split 2.5267 (perplexity 12.5) — against 2.5535 / 2.5690 for the 20B-token model. Training loss fell from 2.69 at the end of the constant phase to 2.62 at the end of the cooldown.

The run was trained with Megatron-Bridge (NeMo 26.04 container) on ALCF Polaris, on 16 nodes × 4 A100-40GB, in about 4.6 hours.

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sfanm/d12-pretrain-climbmix30B-dolmino10B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

This is a base language model and has no chat template. The resumable Megatron distributed checkpoint of the terminal iteration is published under megatron/iter_0076294 (model, optimizer and scheduler state).

This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.