Modelpublic

ashrafs1/nanogpt-3000

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes96downloads

Model Card

nanoGPT-3000

A 124M-parameter GPT-2-class causal language model built from scratch in PyTorch and trained on FineWeb-10B.

This is a pretraining checkpoint (3,000 steps, ~1.57B tokens).

Quick Stats


Parameters	~124M
Context	2,048 tokens
Vocab	50,304 (GPT-2 BPE, padded to 128)
Position enc	RoPE
Norm	RMSNorm + QK-Norm
Activation	Squared ReLU
Attention	FlashAttention-2 varlen (packed docs)
Skips	U-Net style
Optimizer	Muon (2D weights) + AdamW (embeddings)
Batch size	524,288 tokens/step
Hardware	2× A100 40GB
Time	~70 min
MFU	70%
Throughput	375K tok/s

WikiText-2

Model	Tokens	PPL
OpenAI GPT-2	~40B	25.2
nanoGPT-3000	~1.57B	40.50

How to Use

Repo (training code + inference): https://github.com/shehab-ashraf/nanogpt Blog (deep dive): https://shehab-ashraf.github.io/posts/nanogpt/ Logs (W&B): https://wandb.ai/ashrafshehab-/nanoGPT