Modelpublic

geodesic-research/nemotron-super-120b-cc-mt-replay-only-1b-200k-sft

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes20downloads

Model Card

nemotron-super-120b-cc-mt-replay-only-1b-200k-sft

Nemotron 3 Super 120B-A12B non-reasoning (direct-answer chat) SFT from the constitutional-curriculum midtraining study — the SFT stage applied to the cc-mt-replay-only-1b midtrained base.

Training

—Base: geodesic-research/nemotron-super-120b-cc-mt-replay-only-1b (re-imported to Megatron, fresh optimizer warm-start)
—Data: geodesic-research/sft-warm-start-200k no_think
—Recipe: 246 iterations, GBS 128, seq 8192, lr 5e-6 cosine w/ 10% warmup, tokenizer geodesic-research/nemotron-instruct-tokenizer, answer-only loss. Final lm loss 0.6251.
—Topology: TP=1 EP=4 PP=22 ETP=1, 22 nodes / 88 GH200 GPUs on Isambard-AI.
—Coherence: 8/8 clean at T=0.6 topp=0.95, full 8192-token budget — [W&B 0y1llyga](https://wandb.ai/geodesic/megatronbridgeconversioncoherance_tests/runs/0y1llyga)

Usage notes

Chat model — apply the bundled chat template. Recommended sampling: temperature 0.6, top_p 0.95 (Nemotron recommendation; higher temperatures occasionally emit a stray token right before EOS). Loads with native transformers NemotronHForCausalLM; tokenizer PreTrainedTokenizerFast.