grugging/gpt-j-6b-grug

GPT-J Grug

Full finetune of EleutherAI/gpt-j-6B on DarwinAnim8or/grug (38k greentext rows). Trained on Apple Silicon with MLX, AdamW: round 1, ~5.8 epochs at LR 1e-5; round 2 continued 5 epochs at LR 5e-5, batch 8. All 28 transformer blocks updated, embeddings/head kept from base.

Grug speak in short simple words. Sometimes bunga.