CoolFace
Modelpublic

lilob/nanosarvam-owt-run

sourceHugging Facemitupdated 6mo agoView on Hugging Face
0likes5downloads
Model Card

nanosarvam — step 115500

A small GPT-2-style language model trained on OpenWebText.

Architecture

  • —Parameters: 48.28M
  • —Layers: 8
  • —d_model: 512
  • —Heads: 8 (GQA with 2 KV heads)
  • —Context: 512 tokens
  • —FFN: SwiGLU (ff_mult=4)
  • —Positional encoding: RoPE
  • —Training step: 115500
  • —Dataset: Skylion007/openwebtext

Usage

python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("lilob/nanosarvam-owt-run")
model     = AutoModelForCausalLM.from_pretrained("lilob/nanosarvam-owt-run")

inputs = tokenizer("Once upon a time", return_tensors="pt")
out    = model.generate(**inputs, max_new_tokens=100, temperature=0.8, do_sample=True)
print(tokenizer.decode(out[0], skip_special_tokens=True))