jekunz/smollm-135m-cpt-fineweb-faroese-transfer-from-icelandic

This is a SmolLM2-135M-Instruct model fine-tuned first on the Icelandic and then on the Faroese portion of Fineweb-2. It is intended for my research and has not been evaluated more broadly yet.

Training:

• —1 Epoch Icelandic, 5 Epochs Faroese
• —Learning rate: 5e-4
• —LR scheduler: Cosine
• —Warmup ratio: 0.05
• —Batch size: 1
• —4 A100 (40GB) GPUs
• —Gradient accumulation steps: 64
• —Effective batch size: 256
• —Max. context length: 8192 tokens