geodesic-research/nemotron-super-120b-cc-mt-replay-only-1b

nemotron-super-120b-cc-mt-replay-only-1b

Nemotron 3 Super 120B-A12B continued-pretraining (midtraining) checkpoint from the constitutional-curriculum midtraining study (cc-mt).

Training

• —Base checkpoint: NVIDIA-Nemotron-3-Super-120B-A12B-Base-Chat-Init-BF16 (pure Base weights; 1,188 chat-scaffolding embedding rows grafted from Instruct. No post-training behavior included.)
• —Data: 1.0B tokens of pretraining replay only (geodesic-research/Nemotron-Pretraining-Specialized); no constitutional data (compute-matched control)
• —Data order: standard shuffled
• —Recipe ("dyad-1" Base-CPT stability stack): lr 1e-6 cosine w/ 10% warmup, FP32 optimizer states, GBS 128, seq 8192, BF16, 954 iterations = 1.000B tokens (1 epoch). Tokenizer: geodesic-research/nemotron-base-tokenizer (EOD=</s>=id 2). All data filtered against the 1,188 zero-embedding token ids of Super-Base (0 docs dropped).
• —Topology: TP=1 EP=4 PP=22 ETP=1 (parallel folding), 22 nodes / 88 GH200 GPUs on Isambard-AI.
• —W&B: geodesic/megatron_training/462qxe0n

Usage notes

Base-style model (pretraining-format CPT; no instruction tuning) — use completion-style prompting. Loads with native transformers NemotronHForCausalLM; tokenizer PreTrainedTokenizerFast. Coherence-checked post-export (8/8 non-empty, fluent completions).

SFT companions: nemotron-super-120b-cc-mt-replay-only-1b-sr-sft (reasoning) and nemotron-super-120b-cc-mt-replay-only-1b-200k-sft (non-reasoning).