Modelpublic

nikitastheo/phonebabylm-text-deu-eng-sequential_interleaved-dummy

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes64downloads

Model Card

nikitastheo/phonebabylm-text-deu-eng-sequential_interleaved-dummy

Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

—Base config: gpt_dummy_config.json
—Tokenizer: nikitastheo/phonelm-text-deu-eng-tokenizer
—Max steps: 12
—Learning rate: 0.0001
—LR scheduler: linear
—Warmup steps: 16
—Batch size (per device): 32
—Gradient accumulation steps: 1
—Total train batch size: 32
—Language switch steps: 6