SlitherCode/tiny-edu-166m

1---2language: en3license: mit4tags:5  - pretrained6  - causal-lm7  - fineweb-edu8  - custom-architecture9---10 11# tiny-edu-166m (ParchmentLM)12 13A 166M parameter transformer pretrained from scratch on 4B tokens of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).14 15## Architecture (ParchmentLM)16 17Custom decoder-only transformer:18- **Parameters:** 166M19- **Layers:** 1220- **Hidden size:** 76821- **Attention heads:** 1222- **FFN:** SwiGLU (hidden=2048)23- **Context length:** 102424- **Positional encoding:** RoPE (base=10000)25- **Normalization:** RMSNorm26- **Tokenizer:** cl100k_base (100277 tokens) — same as GPT-427 28## Training29 30- **Dataset:** FineWeb-Edu 10BT sample31- **Tokens seen:** ~4B32- **Steps:** 30,00033- **Optimizer:** AdamW (lr=3e-4, cosine decay to 3e-5)34- **Hardware:** Single A100 40GB35 36## Installation37 38```bash39pip install transformers tiktoken40```41 42> **Note:** `tiktoken` is required because the tokenizer wraps OpenAI's cl100k_base encoding43> to guarantee byte-identical token IDs to the vocabulary the model was trained on.44 45## Usage46 47```python48from transformers import AutoTokenizer, AutoModelForCausalLM49 50tokenizer = AutoTokenizer.from_pretrained("SlitherCode/tiny-edu-166m", trust_remote_code=True)51model     = AutoModelForCausalLM.from_pretrained("SlitherCode/tiny-edu-166m", trust_remote_code=True)52 53inputs = tokenizer("The history of mathematics", return_tensors="pt")54out    = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.8, top_k=50)55print(tokenizer.decode(out[0], skip_special_tokens=True))56```57 58## License59 60Model weights: MIT. 61Training data: This work uses the FineWeb-Edu dataset, available under the Open Data Commons Attribution License (ODC-By 1.0).62