Modelpublic

helloboy91/tiny-english-30m

sourceHugging Facemitupdated 9d agoView on Hugging Face

0likes530downloads

Model Card

Tiny English 30M — v0.2.0

An experimental 29.9M-parameter decoder-only model for simple English story completion. It was trained from scratch on a prepared TinyStories subset. It is not an instruction-tuned assistant, factual reference, or production system.

This release replaces v0.1.0 with the checkpoint selected from the additional 100M-token stage. A later 200M-token stage overfit the fixed corpus and is deliberately not released.

Contents and use

model.safetensors contains FP32 inference weights. config.json, tokenizer.json, training_summary.json, evaluation.json, and SHA256SUMS record architecture, provenance, scores, and integrity data. The bundled tiny_english/ package is standalone inference code; optimizer state, raw/cleaned stories, training logs, and local paths are excluded.

bash

python -m pip install -r requirements.txt
python -m tiny_english.generate --model-dir . --prompt "Once upon a time, a little rabbit found a blue hat." --max-new-tokens 100 --temperature 0.7 --seed 42

This is a custom PyTorch export, not a Transformers AutoModelForCausalLM model. Review the bundled code before use.

Architecture

Setting	Value
Transformer blocks / hidden width	8 / 512
Attention	8 causal heads, head width 64
Feed-forward	SwiGLU, intermediate width 1,408
Context	1,024 tokens
Vocabulary	8,192 byte-level BPE tokens
Positions / normalization	RoPE / RMSNorm
Embeddings	Tied input/output

Training and evaluation

The selected checkpoint represents 148,910,080 processed tokens: 54,702,080 prior tokens plus 94,208,000 tokens from the selected additional-training stage checkpoint (step 23,000). Training used 4,096 tokens/update, AdamW, a peak learning rate of 0.0003, and CUDA BF16 on an RTX 3060 Ti for the later stages.

Full nonoverlapping held-out evaluation:

Split	Predicted tokens	Loss	Perplexity
Validation	105,075	1.780266	5.931436
Test	102,850	1.846464	6.337368

The training corpus was prepared from roneneldan/TinyStories at revision f54c09fd23315a6f9c86f9dc80f725de7d8f9c64. It has 96,704 training stories (20,097,225 packed tokens), with deterministic held-out validation and test splits. The tokenizer was trained only on the training split.

TinyStories is synthetic short-story data generated using GPT-3.5 and GPT-4. The source dataset is licensed CDLA-Sharing-1.0; this model release is MIT only to the extent of the author’s rights. See DATA_LICENSE.md.

Limitations

The model can repeat, drift, or produce implausible story continuations. It has no instruction tuning, retrieval, tool use, or factual reliability guarantees. Do not use its output for consequential decisions.