jekunz/smollm-360m-cpt-fineweb-faroese

This is a SmolLM2-350M-Instruct model fine-tuned on the Faroese portion of Fineweb-2. It is intended for my research and has not been evaluated more broadly yet.

Training:

• —5 Epochs
• —Learning rate: 5e-4
• —LR scheduler: Cosine
• —Warmup ratio: 0.05
• —Batch size: 1
• —4 A100 (80GB) GPUs
• —Gradient accumulation steps: 32
• —Effective batch size: 128
• —Max. context length: 8192 tokens