Modelpublic

Manushi0304/tinystories-slm-rope

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes39downloads

Model Card

TinyStories Small Language Model (RoPE + SwiGLU)

A ~30M parameter language model trained from scratch on the TinyStories dataset.

Architecture

Component	Choice	Why
Position Encoding	RoPE	Used in LLaMA, Mistral, Gemma — better length generalization
Activation	SwiGLU	Used in LLaMA, PaLM — better gradient flow than GeLU
Normalization	RMSNorm	Used in LLaMA — faster than LayerNorm
Attention	Flash Attention (PyTorch 2.0+)	Memory efficient causal attention

Parameters

—Total: 29.92M
—Layers: 6 | Heads: 6 | d_model: 384
—Context window: 256 tokens

Training

—Dataset: TinyStories (~2.1M short stories)
—Optimizer: AdamW (β₁=0.9, β₂=0.95, weight_decay=0.1)
—LR Schedule: Linear warmup + Cosine decay
—Mixed precision: bfloat16
—Best validation loss: 1.6472 | Perplexity: 5.2

Sample Output

Prompt: "Once upon a time there was a little girl named Lily"

[Add sample output after training]

Usage

python

import torch
import tiktoken
from huggingface_hub import hf_hub_download

# Load weights
weights_path = hf_hub_download(repo_id="Manushi0304/tinystories-slm-rope", filename="pytorch_model.bin")
# Load config and rebuild model, then load state dict