atrost/climbmix-matformer-353m-1p2b-h100

atrost/climbmix-matformer-353m-1p2b-h100

MatFormer-style nested Llama pretrained from scratch on nvidia/Nemotron-ClimbMix.

Training details

• —Architecture: MatFormerForCausalLM
• —XL parameters: 352,943,360
• —Context length: 2048
• —Target tokens: 1,200,000,000
• —Actual tokens: 1,201,668,096
• —Tokens per optimizer step: 2,097,152
• —Size sampling: uniform over S/M/L/XL, one size per optimizer step
• —Optimizer: AdamW
• —LR schedule: cosine decay with warmup
• —Precision: bfloat16

MatFormer submodels

• —S: 159,776,000 params, {'numattentionheads': 2, 'numkeyvalueheads': 1, 'intermediatesize': 512}
• —M: 190,741,760 params, {'numattentionheads': 4, 'numkeyvalueheads': 2, 'intermediatesize': 1024}
• —L: 252,673,280 params, {'numattentionheads': 8, 'numkeyvalueheads': 4, 'intermediatesize': 2048}
• —XL: 352,943,360 params, {'numattentionheads': 16, 'numkeyvalueheads': 8, 'intermediatesize': 3584}

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "atrost/climbmix-matformer-353m-1p2b-h100"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype="auto",
)

trust_remote_code=True is required for the custom MatFormer architecture.