nikitastheo/v3-babylm-deu-ell-sequential_interleaved

nikitastheo/v3-babylm-deu-ell-sequential_interleaved

Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

• —Base config: model_configs/gpt_base_config.json
• —Tokenizer: nikitastheo/babylm-deu-tokenizer
• —Max steps: 24960
• —Learning rate: 0.0001
• —LR scheduler: linear
• —Warmup steps: 2496
• —Batch size (per device): 32
• —Gradient accumulation steps: 1
• —Total train batch size: 32
• —Language switch epoch: 10