Azrail/smallm_70

1---2library_name: transformers3license: mit4datasets:5- YourDatasetName/if-applicable6language:7- en8pipeline_tag: text-generation9tags:10- transformer11- language-model12- experimental13---14 15# **SmalLM**16 17<hr>18<div align="center">19  <a href="https://github.com/azrails/SmalLm" target="_blank" style="margin: 2px;">20    <img alt="GitHub" src="https://img.shields.io/badge/GitHub-SmalLM-181717?logo=github" style="display: inline-block; vertical-align: middle;"/>21  </a>22  <a href="https://github.com/azrails/SmalLm/blob/main/LICENSE" style="margin: 2px;">23    <img alt="License" src="https://img.shields.io/badge/License-MIT-blue.svg" style="display: inline-block; vertical-align: middle;"/>24  </a>25</div>26 27SmalLM is a series of small transformer models built from scratch for language modeling. This project is designed to explore innovative approaches to transformer architectures through modular pipelines for pretraining, fine-tuning, and alignment.28 29## Uses30 31```python32from transformers import AutoTokenizer, AutoModelForCausalLM33 34tokenizer = AutoTokenizer.from_pretrained("Azrail/smallm_70")35model = AutoModelForCausalLM.from_pretrained("Azrail/smallm_70", trust_remote_code=True)36inputs = tokenizer("How are you?", return_tensors="pt")37 38out = model.generate(**inputs, max_new_tokens=100)39print(tokenizer.batch_decode(out))40```41 42## Model Details**43**Key Features:**44 451. Grouped Query Attention (GQA).46 472. Mixture-of-Experts with auxiliary loss-free balancing.48 493. ALiBi (Attention with Linear Biases) or Rotary Position Embedding (RoPE).50 514. NTK-by-parts RoPE interpolation for extends context length.52 53**Pre-Training**:54 55| Model                | Training Data                                                                 | Steps | Content Length | Tokens | LR    | Batch Size | Precision |56|----------------------|-------------------------------------------------------------------------------|-------|----------------|--------|-------|------------|-----------|57| [SmalLM-70M](https://huggingface.co/Azrail/smallm_70)      | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)                | 70k    | 1024           | 18B     | 1e-3  | 0.25M       | bfloat16  |58| [SmalLM-150M](#)      | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)                | -   | 1024           | -    | -  | -         | bfloat16  |59| [SmalLM-350M](#)     | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)                | -   | 1024           | -    | -  | -       | bfloat16  |60| [SmalLM-500M](#)     | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)                | -   | 1024           | -    | -  | -         | bfloat16  |61 62**Evaluation**:63Evaluation runing with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)64 65| Model                | MMLU | ARC easy/hard | PIQA  | HellaSwag | OBQA  | Winogrande |66|----------------------|------|----------------|-------|-----------|-------|------------|67| [SmalLM-70M](#)      | 25.33 | 51.47/25.68    | 61.75 | 30.31     | 30.8  | 50.83      |68| [SmalLM-150M](#)     | -    | -              | -     | -         | -     | -          |69| [SmalLM-350M](#)     | -    | -              | -     | -         | -     | -          |70| [SmalLM-500M](#)     | -    | -              | -     | -         | -     | -          |71 72 73**Procedure**:74 75[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://api.wandb.ai/links/azrails-main/58rwb1yb)76 77### Framework versions78 79- Transformers 4.50.380- Pytorch 2.6.0+cu12681- Datasets 3.5.082- Tokenizers 0.21.1