Modelpublic

nikitastheo/phonelm-text-seed46-deu-eng

sourceHugging Faceupdated 16d agoView on Hugging Face

0likes909downloads

Model Card

nikitastheo/phonelm-text-seed46-deu-eng

Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

—Base config: model_configs/gpt_base_config.json
—Tokenizer: nikitastheo/phonelm-text-seed46-deu-eng-tokenizer
—Max steps: 30000
—Learning rate: 0.0001
—LR scheduler: linear
—Warmup steps: 3000
—Batch size (per device): 32
—Gradient accumulation steps: 1
—Total train batch size: 32
—Language switch step: 10000