Azrail/smallm_70

SmalLM

<hr> <div align="center"> <a href="https://github.com/azrails/SmalLm" target="_blank" style="margin: 2px;"> <img alt="GitHub" src="https://img.shields.io/badge/GitHub-SmalLM-181717?logo=github" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://github.com/azrails/SmalLm/blob/main/LICENSE" style="margin: 2px;"> <img alt="License" src="https://img.shields.io/badge/License-MIT-blue.svg" style="display: inline-block; vertical-align: middle;"/> </a> </div>

SmalLM is a series of small transformer models built from scratch for language modeling. This project is designed to explore innovative approaches to transformer architectures through modular pipelines for pretraining, fine-tuning, and alignment.

Uses

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Azrail/smallm_70")
model = AutoModelForCausalLM.from_pretrained("Azrail/smallm_70", trust_remote_code=True)
inputs = tokenizer("How are you?", return_tensors="pt")

out = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.batch_decode(out))

Model Details**

Key Features:

1. 1.Grouped Query Attention (GQA).

1. 1.Mixture-of-Experts with auxiliary loss-free balancing.

1. 1.ALiBi (Attention with Linear Biases) or Rotary Position Embedding (RoPE).

1. 1.NTK-by-parts RoPE interpolation for extends context length.

Pre-Training:

Evaluation: Evaluation runing with lm-evaluation-harness

Procedure:

<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>

Framework versions

• —Transformers 4.50.3
• —Pytorch 2.6.0+cu126
• —Datasets 3.5.0
• —Tokenizers 0.21.1