jonmabe/tiny-llm-demo

Tiny-LLM Text Generator

A 54 million parameter language model trained from scratch on Wikipedia.

About

This demonstrates that meaningful language models can be trained on consumer hardware with modest compute budgets!

Architecture

Training

• —Training Steps: 50,000
• —Tokens: ~100M
• —Hardware: NVIDIA RTX 5090 (32GB)
• —Training Time: ~3 hours

Model

jonmabe/tiny-llm-54m

Limitations

• —Small model size limits knowledge and capabilities
• —Trained only on Wikipedia - limited domain coverage
• —May generate factually incorrect information
• —Not instruction-tuned