nikitastheo/v5-goldfish-500mb-25k-lower-seed44-ell2-ell-sequential_interleaved

nikitastheo/v5-goldfish-500mb-25k-lower-seed44-ell2-ell-sequential_interleaved

Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

• —Base config: model_configs/gpt_base_config.json
• —Tokenizer: nikitastheo/goldfish-500mb-25k-ell-lower-seed44-tokenizer
• —Max steps: 124060
• —Learning rate: 0.0001
• —LR scheduler: linear
• —Warmup steps: 12406
• —Batch size (per device): 64
• —Gradient accumulation steps: 1
• —Total train batch size: 64
• —Language switch epoch: 10