pythonstudentiam/tinyllm

1---2license: cdla-sharing-1.03datasets:4  - roneneldan/TinyStories5  - roneneldan/TinyStoriesInstruct6language:7  - en8pipeline_tag: text-generation9tags:10  - llama11  - tiny12  - educational13  - gguf14---15 16# tinyllm — instruction-tuned17 18A 15.7M-parameter Llama-architecture language model trained from19random initialization on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).20 21Built as a complete walk through the model lifecycle — tokenizer, architecture,22pretraining, evaluation, instruction tuning, packaging, quantization, and local23serving. It is small enough to train in about 45 minutes on a free Colab T4 and24to run on a 2-core laptop CPU with no GPU.25 26## Architecture27 28| | |29|---|---|30| Parameters | 15,735,168 (12,589,440 non-embedding) |31| Layers | 8 |32| Hidden size | 384 |33| Attention heads | 6 query / 2 key-value (GQA) |34| Head dim | 64 |35| MLP | SwiGLU, intermediate 1024 |36| Normalization | RMSNorm (eps 1e-05) |37| Position encoding | RoPE (theta 10000) |38| Context length | 512 |39| Vocabulary | 8192 (SentencePiece BPE, byte fallback) |40| Embeddings | tied input/output |41 42## Training43 44| | |45|---|---|46| Tokens | 164M (~10 per parameter) |47| Steps | 2,500 at 65,536 tokens/step |48| Optimizer | AdamW (betas 0.9/0.95, wd 0.1 on matrices only) |49| Schedule | cosine, 200 warmup steps, peak LR 0.0006 |50| Precision | fp16 AMP with loss scaling |51| Hardware | 1x NVIDIA T4 (Colab free tier) |52 53## Usage54 55```python56from transformers import AutoModelForCausalLM, AutoTokenizer57 58tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm")59model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm")60 61messages = [{"role": "user", "content": "Write a story about a lost puppy."}]62prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)63ids = tok(prompt, return_tensors="pt")64out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8)65print(tok.decode(out[0], skip_special_tokens=True))66```67 68### With llama.cpp69 70GGUF conversions are included in this repo.71 72```bash73llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 808074```75 76## Limitations77 78This model has 15.7M parameters and a 8192-token vocabulary,79trained exclusively on synthetic children's stories. Be concrete about what that means:80 81- **It only does one thing.** It writes simple short stories in the TinyStories82  style. Anything else — code, arithmetic, factual questions, translation,83  summarization of arbitrary text — produces confident nonsense.84- **Its vocabulary is small.** Words outside a children's-story vocabulary fall85  back to individual bytes, which it handles poorly.86- **Context is 512 tokens.** There is no long-range coherence to be had.87- **No safety tuning of any kind.** It has had no alignment work beyond88  instruction tuning on story prompts.89- **Quantization hurts more than usual.** Small models have less parameter90  redundancy to absorb rounding error; Q4_K_M is measurably worse here than the91  usual "negligible loss" guidance for 7B+ models would suggest.92 93Not suitable for any production use. It is a teaching artifact.94 95## Training data96 97[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) — synthetic98short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a993-4 year old. Licensed CDLA-Sharing-1.0.100 101Instruction tuning used [TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct).102 103