nikitastheo/v5-babylm-25k-lower-ind-ell-sequential_interleaved

nikitastheo/v5-babylm-25k-lower-ind-ell-sequential_interleaved

Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

• —Base config: model_configs/gpt_base_config.json
• —Tokenizer: nikitastheo/babylm-25k-ind-lower-tokenizer
• —Max steps: 23340
• —Learning rate: 0.0001
• —LR scheduler: linear
• —Warmup steps: 2334
• —Batch size (per device): 32
• —Gradient accumulation steps: 1
• —Total train batch size: 32
• —Language switch epoch: 10