molfe/everyones-llm-workshop

EveryonesLLM workshop model

This model was trained from scratch in the EveryonesLLM GPU workshop.

Training data

• —Dataset: HuggingFaceFW/fineweb-edu
• —Sample: sample/10BT/000_00000.parquet
• —Tokenizer: tiktoken GPT-2

Model

• —Embedding dimension: 512
• —Hidden dimension: 2048
• —Attention heads: 8
• —Transformer blocks: 4
• —Context length: 512

Training

• —Batch size: 64
• —Steps: 2000
• —Learning rate: 0.001
• —BF16: True
• —Seen tokens: 65536000
• —Final validation loss: 4.865565538406372

Loading

This is a custom educational PyTorch model, not a Transformers AutoModel package. Use the EveryonesLLM workshop source code to instantiate nanoGPT, then load model.pt.

Notes

The code originates from the MIT-licensed EveryonesLLM project. FineWeb-Edu is distributed under ODC-By. Check the dataset and repository terms before redistributing derived artifacts.