CoolFace
Modelpublic

AxiomicLabs/GPT-S-5M

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face
17likes319downloads
Model Card

[image]

GPT-S-5M

GPT-S-5M is a first-generation model in the GPT-S small-model family: 5M parameters, 25B training tokens, a custom 4K tokenizer, 9 layers, and our all new T-X3 architecture, trained from scratch on a 3-source corpus.

At 5M parameters, GPT-S-5M achieves best-in-class performance among the small open models evaluated in this benchmark set, outperforming models 6x larger.

Benchmarks

All models evaluated in bf16 with an internal harness modeled on EleutherAI/lm-eval-harness. Scores are zero-shot; normalized accuracy is used where available.

CompanyModelHellaswagARC (easy)PIQAArithmarkBLiMPAverage
Axiomic LabsGPT-S-5M27.39%33.16%57.13%31.5%72.21%44.28%
EleutherAIpythia-31m27.14%33.88%56.26%29.44%67.78%42.90%
EleutherAIpythia-14m26.20%32.28%55.88%28.06%66.75%41.83%
LH-Tech-AISpark-5M-Base-v427.03%33.21%53.43%32.70%62.17%41.71%
SupraLabsSupra-Mini-v5-8M26.38%33.33%54.03%27.30%63.83%40.97%
SupraLabsSupra-Mini-v4-2M25.52%30.98%51.90%29.72%60.57%39.74%

Architecture

<a href="https://hfviewer.com/AxiomicLabs/GPT-S-5M?utmsource=huggingface&amp;utmmedium=embeddedmodelcard&amp;utmcampaign=AxiomicLabsGPT-S-5Mcard" target="_blank" rel="noopener"> <img src="https://hfviewer.com/api/card.svg?source=AxiomicLabs%2FGPT-S-5M&amp;granularity=auto&amp;v=20260516-title-pills-card" alt="Open AxiomicLabs/GPT-S-5M in hfviewer" width="100%" /> </a>

ComponentDetails
Position encodingRoPE, theta=2,500
NormalizationRMSNorm
Feed-forwardSwiGLU
AttentionExclusive Grouped-query attention (XGQA), 6 query heads / 2 KV heads
EmbeddingsWeight tied
Context length512 tokens
Parameters5,158,464

Config

text
vocab_size       = 4,096
hidden_size      = 192
num_layers       = 9
num_heads        = 6
num_kv_heads     = 2
head_dim         = 32
intermediate     = 672
block_size       = 512
rope_theta       = 2,500

Training

GPT-S-5M was trained for 25B tokens with a mixture built around educational web text, synthetic textbook-style material, and higher-quality web text.

SourceDatasetMixPurpose
FineWeb-EduHuggingFaceFW/fineweb-edu55%Primary educational web text
Cosmopedia v2HuggingFaceTB/smollm-corpus30%Synthetic textbook-style coverage
FineWeb-HQepfml/FineWeb-HQ15%Higher-quality general web text

Hyperparameters

HyperparameterValue
OptimizerAdamW
Adam betas0.9 / 0.95
Weight decay0.01
Peak learning rate2.5e-3
Minimum learning rate0
LR scheduleWarmup-stable-decay
Warmup steps1,500
Decay start70% of training
Training tokens25B
Total batch size262,144 tokens
Microbatch128 x 512 tokens
Gradient accumulation steps4
Gradient clipping1.0
Precisionbfloat16 autocast

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "AxiomicLabs/GPT-S-5M"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.1,
        no_repeat_ngram_size=4,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is a small base language model. It is not instruction tuned, has limited factual capacity, and uses a 512 token context window.