linkanjarad/Gouda-50M-SFT

<h1 align="center" style="color: gold"> Gouda-50M-SFT </h1>

I've always wanted to train a large language model from scratch. Most tutorials and walkthroughs start with the original transformer paper from 2017, "Attenttion is All You Need". However, I wanted to learn the newer advancements in LLMs, such as Grouped Query Attention, SwiGLU, RoPE, and Tied Embeddings. And that is what I did! I coded the model architecture from scratch in raw Pytorch (no Huggingface), and came up with a model which I named Gouda. Gouda-50M-SFT is a Llama-style transformer trained from scratch on only 1 billion training tokens (spanning books, wikipedia, github, and reddit) and 1 billion SFT tokens.

Gouda-50M-SFT is a 50 million parameter decoder-only language model trained from scratch using a Llama-style architecture. The model adopts modern architectural components including Grouped Query Attention (GQA), Rotary Position Embeddings (RoPE), SwiGLU feed-forward layers, RMSNorm, and tied input/output embeddings with a 1k context length.

The model is designed as an efficient small-scale research model suitable for experimentation, educational purposes, lightweight inference, and studying scaling behavior in modern language models.

<p align="center"> <img src="images/eval.jpg" width="600"> </p>

Model Details

• —Architecture: Decoder-only Transformer
• —Parameters: 54.2 million
• —Context Length: 1k
• —Vocabulary Size: 32,000
• —Tokenizer: unsloth/llama-2-7b
• —Embedding Dimension: 512
• —Transformer Layers: 12
• —Attention Heads: 16
• —Key/Value Heads: 8 (Grouped Query Attention)
• —Feed-Forward Dimension: 1536 (SwiGLU)
• —Dropout: 0.1

Architecture

Gouda-50M-SFT follows a modern Llama-style decoder architecture:

• —Rotary Position Embeddings (RoPE)
• —Grouped Query Attention (GQA)
• —RMSNorm pre-normalization
• —SwiGLU feed-forward networks
• —Tied token embeddings and language modeling head

Each transformer block consists of:

1. 1.RMSNorm
2. 2.Multi-head grouped-query self-attention
3. 3.Residual connection
4. 4.RMSNorm
5. 5.SwiGLU feed-forward network
6. 6.Residual connection

The architecture is intentionally lightweight while retaining many of the design choices used in contemporary large language models.

Training

Training Datasets

Optimization

• —Optimizer: AdamW
• —Learning Rate: 1e-4 peak
• —Minimum Learning Rate: 1e-5
• —Learning Rate Schedule: Cosine decay with linear warmup
• —Warmup Steps: 2,000
• —Weight Decay: 0.1
• —Gradient Clipping: 1.0
• —Adam β1: 0.9
• —Adam β2: 0.95

Training Configuration

• —Sequence Length: 1024
• —Micro Batch Size: 12
• —Gradient Accumulation Steps: 8
• —Effective Batch Size: 96 sequences
• —Training Steps: 50K for base, 27.5K for SFT
• —GPU: RTX 5080

Limitations

As a 50M parameter language model, Gouda-50M-SFT has limited reasoning ability, factual knowledge, and long-context capabilities compared to larger models. Outputs may contain inaccuracies, hallucinations, or incomplete information.

Performance varies substantially depending on prompt format and task complexity.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("linkanjarad/Gouda-50M-SFT")
model = AutoModelForCausalLM.from_pretrained("linkanjarad/Gouda-50M-SFT")

user_input = "What is Artificial Intelligence?"
prompt = f"<|turn>user\n{user_input}\n<turn|>\n<|turn>model\n"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    top_k=64,
    top_p=0.9,
    temperature=0.7
)

print(tokenizer.decode(outputs[0]))