Modelpublic

OptimizerStudy/adamw_1.2b_8

sourceHugging Faceupdated 11mo agoView on Hugging Face

0likes31downloads

Model Card

Model Card

—Source: https://arxiv.org/abs/2509.02046
—Optimizer: adamw
—Model size: 1.2b
—Data size: 193B

Best configuration

Hyperparameter	Value
beta1	`0.9`
beta2	`0.98`
epsilon	`1e-10`
learning_rate	`0.002`
maxgradnorm	`1`
minlrratio	`0.0`
nesterov	`False`
trainbatchsize	`256`
warmup	`1000`
weight_decay	`0.2`