nikitastheo/v2-dummy-babylm-deu-ell-sequential_interleaved

nikitastheo/v2-dummy-babylm-deu-ell-sequential_interleaved

Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

• —Base config: model_configs/gpt_dummy_config.json
• —Tokenizer: nikitastheo/babylm-deu-tokenizer
• —Max steps: 12
• —Learning rate: 0.0001
• —LR scheduler: linear
• —Warmup steps: 6
• —Batch size (per device): 32
• —Gradient accumulation steps: 1
• —Total train batch size: 32
• —Language switch epoch: 3