Modelpublic

OptimizerStudy/muon_130m_4

sourceHugging Faceupdated 11mo agoView on Hugging Face

0likes268downloads

Model Card

Model Card

—Source: https://arxiv.org/abs/2509.02046
—Optimizer: muon
—Model size: 130m
—Data size: 10B

Best configuration

Hyperparameter	Value
beta1	`0.8`
beta2	`0.98`
decay	`0.8`
epsilon	`1e-15`
learning_rate	`0.008`
lr_schedule	`linear`
maxgradnorm	`1`
minlrratio	`0`
momentum	`0.95`
muon_epsilon	`1e-05`
muontoadam_lr	`0.3`
trainbatchsize	`128`
warmup	`0`
weight_decay	`0.1`