AxiomicLabs/GPT-X-125M

[image]

GPT-X-125M

A modern Llama-style language model trained from scratch based on the T-X1 architecture. 125M parameters, 15B tokens of FineWeb-Edu. Outperforms GPT-3 (125M) on HellaSwag using 20x less training data.

Results

Evaluated with an internal harness modeled on EleutherAI/lm-eval-harness; all benchmarks are zero-shot.

**LogicMark** and **ArithMark** are procedural benchmarks designed to evaluate structured reasoning and arithmetic generalization across increasing difficulty levels.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Datdanboi25/GPT-X-125M",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("Datdanboi25/GPT-X-125M")

inputs = tokenizer("The future of AI is", return_tensors="pt", add_special_tokens=True)
output = model.generate(
    **inputs, 
    max_new_tokens=50, 
    do_sample=True, 
    temperature=0.8,
    pad_token_id=tokenizer.eos_token_id,
    repetition_penalty=1.1
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Architecture

GPT-X replaces every major component of GPT-2 with modern alternatives proven at scale by Llama 3, Mistral, and Gemma 2.

Config

vocab_size     = 50,304    (GPT-2 BPE, padded)
n_layer        = 27
n_head         = 9         (query heads)
n_kv_heads     = 3         (key-value heads, 3:1 GQA)
n_embd         = 576
head_dim       = 64
intermediate   = 1,536     (SwiGLU, 2.67x ratio)
block_size     = 1,024
rope_theta     = 10,000
total params   = 124,561,728

Parameter Breakdown

Training

Data

• —Dataset: FineWeb-Edu sample-100BT (educational web text)
• —Tokens: 15B (30,500 steps x 524,288 tokens/step)
• —Tokenizer: GPT-2 BPE (tiktoken, 50,257 vocab padded to 50,304)

Optimization

• —Optimizer: AdamW (betas=0.9/0.95, weight_decay=0.1)
• —Learning rate: 6e-4 max, 6e-5 min
• —Schedule: WSD — 1,000 step warmup, stable phase, linear decay over final 20%
• —Batch size: 524,288 tokens (microbatch=8, seqlen=1024, grad accumulation)
• —Precision: bfloat16 mixed precision
• —Gradient clipping: 1.0

Hardware

• —1x NVIDIA RTX 3080 Ti (training)
• —1x Intel i9-13900K (data tokenization)
• —Training time: ~100 hours

Design Decisions

• —27 layers x 576 hidden — SmolLM-135M and MobileLLM-125M proved deep & narrow is SotA at 125M scale. 2.25x more depth than GPT-2's 12 layers.
• —GQA 3:1 — Saves attention parameters reinvested into a larger SwiGLU. Negligible quality loss at this ratio.
• —SwiGLU — Gated MLP with SiLU outperforms GELU across Llama, PaLM, and Mistral.
• —QK-Norm — Prevents attention logit explosion in deep models. Applied before RoPE (Llama 3.1 / Gemma 2 ordering).
• —z-loss — Prevents logit magnitude drift during training (PaLM, T5).
• —WSD schedule — Holds at peak LR for 80% of training, then sharp decay. Beats cosine with limited tokens.
• —No bias — Zero quality benefit in modern transformers. Confirmed by every post-2023 frontier LLM.

Limitations

• —Small model: 125M parameters limits reasoning and factual recall
• —Educational data only: Trained on FineWeb-Edu; not representative of general web text
• —Not instruction-tuned: Base model only, not aligned for chat
• —English only
• —1024 context window

License

Apache 2.0

Citation

@misc{gptx2025,
  title={GPT-X: A Modern Llama-Style Language Model at 125M Scale},
  author={Datdanboi25},
  year={2025},
  url={https://huggingface.co/Datdanboi25/GPT-X-125M}
}