CorticalStack/gemma-7b-ultrachat-sft

gemma-7b-ultrachat-sft

gemma-7b-ultrachat-sft is an SFT fine-tuned version of google/gemma-7b using the stingning/ultrachat dataset.

Fine-tuning configuration

LoRA

• —LoRA r: 8
• —LoRA alpha: 16
• —LoRA dropout: 0.1

Training arguments

• —Epochs: 1
• —Batch size: 4
• —Gradient accumulation steps: 6
• —Optimizer: pagedadamw32bit
• —Max steps: 100
• —Learning rate: 0.0002
• —Weight decay: 0.001
• —Learning rate scheduler type: constant
• —Max seq length: 2048