Oxpose/Soma-1M

SRLM-1M

A Small ROSA based Language Model, of 900k parameters.

Training inforation

Trained on smollm-corpus fineweb-edu-dedup subset, over 16M tokens.

Evaluations

• —Wikitext v2 byte_perplexity: 4.4553
• —Arceasy accnorm: 28.66%
• —Arc_easy acc: 26.39%
• —Blimp acc: 53.1%
• —Hellaswag acc: 26.65%
• —Hellaswag acc_norm: 27.07%

How to run it

To run this model, you have to:

1. 1.Download the files
2. 2.Compile rosa.pyx using python setup.py build_ext --inplace
3. 3.Load the model with the correct configuration
4. 4.Call the model