Fu01978/TinyLM

1---2language: en3license: mit4tags:5- tiny6- language-model7- causal-lm8- pytorch9datasets:10- roneneldan/TinyStories11- Skylion007/openwebtext12pipeline_tag: text-generation13library_name: transformers14---15 16# TinyLM17 18A 3.4M parameter causal language model trained from scratch, for experimentation.19 20## Architecture21 22| Hyperparameter | Value |23|---|---|24| Parameters | 3.403.968 |25| Layers | 4 |26| Hidden size | 64 |27| Attention heads | 4 |28| FFN dim | 192 |29| Embedding rank | 32 |30| Context length | 256 |31| Tokenizer | GPT-2 (50257 vocab) |32 33Uses a **factored (low-rank) embedding** to keep the vocab projection from eating the entire parameter budget, with weight tying on the output head.34 35## Training36 37| | |38|---|---|39| Datasets | Skylion007/openwebtext (10k samples), roneneldan/TinyStories (10k samples) |40| Optimizer | AdamW (lr=3e-3, weight_decay=0.01) |41| Scheduler | Cosine annealing with warm restarts |42| Mixed precision | fp16 (torch.cuda.amp) |43| Hardware | Nvidia P100 |44 45## Usage46```python47from huggingface_hub import snapshot_download48import importlib.util49import torch50 51# Download files52snapshot_download(repo_id="Fu01978/TinyLM", local_dir="./tinylm")53 54# Load via script55spec   = importlib.util.spec_from_file_location("modeling_tinylm", "./tinylm/modeling_tinylm.py")56module = importlib.util.module_from_spec(spec)57spec.loader.exec_module(module)58 59model, tokenizer, config = module.load_tinylm("./tinylm")60model.eval()61 62# Generate63output = module.generate(model, tokenizer, "Once upon a time, ")64print(output)65```