Modelpublic

rkstgr/nanochat-d24-speedrun

sourceHugging Faceupdated 6mo agoView on Hugging Face

2likes

Model Card

nanochat-d24-speedrun

A 1.4B parameter GPT-2 style model trained from scratch using nanochat on 16×H100 GPUs.

Training

—Architecture: 24-layer Transformer, 1536 hidden dim, 12 heads, 32K vocab
—Training data: 5.8B tokens (ClimbMix), param:data ratio = 8
—Precision: FP8 (tensorwise scaling)
—Hardware: 16× NVIDIA H100
—Throughput: 1.58M tok/sec, 47.6% bf16 MFU
—Pretraining time: 62 minutes
—Total pipeline time: 1h 26m (pretrain + base eval + SFT + chat eval + report)

Results

Metric	Base	SFT
Val BPB	0.715	-
CORE	0.247	-
ChatCORE	-	0.360
ARC-Easy	-	61.3%
ARC-Challenge	-	48.9%
MMLU	-	36.4%
HumanEval	-	11.0%
GSM8K	-	9.8%
SpellingBee	-	99.6%

Files

—base_checkpoints/ — Pretrained base model (step 5568)
—chatsft_checkpoints/ — SFT fine-tuned chat model (step 482)
—tokenizer/ — Custom BPE tokenizer (32K vocab)
—report.md — Full training report

Usage

python

# Requires the nanochat repo
from nanochat.gpt import GPT, GPTConfig
from nanochat.checkpoint_manager import load_checkpoint

model, metadata = load_checkpoint("path/to/base_checkpoints")

Acknowledgments

Compute resources provided by WestAI. Thanks to the WestAI team for their generous compute contributions.