pythonstudentiam/tinyllm

tinyllm — instruction-tuned

A 15.7M-parameter Llama-architecture language model trained from random initialization on TinyStories.

Built as a complete walk through the model lifecycle — tokenizer, architecture, pretraining, evaluation, instruction tuning, packaging, quantization, and local serving. It is small enough to train in about 45 minutes on a free Colab T4 and to run on a 2-core laptop CPU with no GPU.

Architecture

Training

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm")
model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm")

messages = [{"role": "user", "content": "Write a story about a lost puppy."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

With llama.cpp

GGUF conversions are included in this repo.

llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 8080

Limitations

This model has 15.7M parameters and a 8192-token vocabulary, trained exclusively on synthetic children's stories. Be concrete about what that means:

• —It only does one thing. It writes simple short stories in the TinyStories style. Anything else — code, arithmetic, factual questions, translation, summarization of arbitrary text — produces confident nonsense.
• —Its vocabulary is small. Words outside a children's-story vocabulary fall back to individual bytes, which it handles poorly.
• —Context is 512 tokens. There is no long-range coherence to be had.
• —No safety tuning of any kind. It has had no alignment work beyond instruction tuning on story prompts.
• —Quantization hurts more than usual. Small models have less parameter redundancy to absorb rounding error; Q4KM is measurably worse here than the usual "negligible loss" guidance for 7B+ models would suggest.

Not suitable for any production use. It is a teaching artifact.

Training data

TinyStories — synthetic short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a 3-4 year old. Licensed CDLA-Sharing-1.0.

Instruction tuning used TinyStoriesInstruct.