CoolFace
Modelpublic

FlameF0X/NanoGPT2-FP128

sourceHugging Facemitupdated 6mo agoView on Hugging Face
0likes13downloads
Model Card

nano-gpt2-fp128

A nano GPT-2 style causal language model trained on TinyStories with double-double (~FP128) arithmetic in the forward pass.

Architecture

Hyper-parameterValue
Embedding dim32
Attention heads2
Transformer layers2
Context window64
Vocabulary82 (char-level, TinyStories)
Parameters32,768

Precision

StagePrecision
Weight storagefloat64
Forward matmuls~106-bit (double-double / Veltkamp split)
Backward passfloat64
Equivalent toIEEE binary128 (113-bit) within 7 bits