rdj-034/lab2_efficient

lab2_efficient

Hyperparameters

• —learning_rate: 2e-5
• —perdevicetrainbatchsize: 128
• —effectivebatchsize: 128
• —gradientaccumulationsteps: 1
• —weight_decay: 0.1
• —optimizer: adamw_torch
• —fp16: True
• —gradient_checkpointing: True
• —lr_scheduler: cosine
• —warmup_ratio: 0.1
• —max_steps: 100

Results