Guspard-ew/BeanSLM-Instruct-278M

BeanSLM-Instruct-278M

BeanSLM-Instruct-278M is a 278M parameter English fully tuned language model trained from scratch.

Overview

This model was trained on a sequence length of 256 using a mixed dataset of about 130k instruction examples. It is meant for text generation and simple instruction following.

Training Notes

• —Pretraining dataset: FineWebEdu
• —Instruction data: diverse130kbetter.txt
• —Vocabulary size: 32k
• —Sequence length: 256
• —Training status: trained from scratch
• —Final pretraining loss: 3.3
• —Final instruct loss: 1.7 (i dont know if this version could be overfit, i thought 1.7 was too low but after testing it didnt seem overfit)

What this model is good at

• —Short instruction following
• —Simple assistant text generation
• —Lightweight local experiments

Limitations

• —Small model size
• —Short context length
• —Will struggle with logic math and complex reasonong (bad at math !)