Modelpublic

lemonteaa/nanogpt-speedrun

sourceHugging Faceupdated 2y agoView on Hugging Face

3likes

Model Card

NanoGPT Speedrun

Following https://github.com/KellerJordan/modded-nanogpt for fun (learning).

Run Info

baseline/

—Run on lightning cloud, using one L40S
—Batch size set to 32
—VRAM usage: 26.95GB (25698MB reported in nvidia-smi)
—4 seconds per step, total 3200 steps
—Checkpoint saved every 320 steps

Training loss

To experimentally check the neural scaling law:

[image]

(Fitted line: log y = -0.11 * log x + 0.9 where x is step (0 to 3200) and y is the training loss)

Demo

Available at https://huggingface.co/spaces/lemonteaa/nanogpt-speedrun-demo

(WIP)