Modelpublic

geodesic-research/nemotron_120b_warm_start_sft_200k_instruct

sourceHugging Faceotherupdated 5mo agoView on Hugging Face

1likes23downloads

Model Card

Nemotron-3 120B-A12B — Warm-Start SFT 200k (instruct)

This is one of four canonical warm-start baselines for the Geodesic Research SFM / inoculation campaigns. Models trained with this checkpoint as the starting point should be directly comparable across the {30B, 120B} × {think, instruct} matrix.

Variant

`instruct` — uses the `geodesic-research/nemotron-instruct-tokenizer`, whose chat template never auto-injects <think>...</think> reasoning tags. Inference produces direct instruct-style responses without reasoning traces..

The encoder is byte-identical to the upstream `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16` tokenizer; only the chat template differs from upstream.

Training data

—HF dataset: `geodesic-research/sft-warm-start-200k`
—Subset: no_think
—Examples: 200,000 chat-format conversations (no held-out validation/test split)
—Tokens: 259M (counted with the instruct tokenizer)
—Sequence packing: pad_seq_to_mult=1, pad_to_max_length=false, packed sequence length 8192

Training recipe


Hardware	Isambard GH200, 16 nodes × 4 GPUs (sm_90, BF16)
Parallelism	TP=4, EP=4, PP=8, CP=1, ETP=1 (parallel folding)
Data parallelism	DPpure=2, gradaccum=32
Global batch size	64 (× 8192 tokens = 524k tokens/batch)
Sequence length	8192
Optimizer	distributed Adam (β1=0.9, β2=0.95, ε=1e-8, wd=0.1, clip=1)
Learning rate	5e-06 (cosine decay to 0, 5% warmup)
Precision	BF16 + Precision-Aware Optimizer (BF16 momentum/variance) + activation offloading
Iterations	495 (≈ 1 epoch over the dataset)
W&B run	link