Beebey/smallcoder-303m

🧠 SmallCoder (303M)

SmallCoder is a 303M parameter LLaMA-style language model trained from scratch for code generation and algorithmic reasoning.

This checkpoint represents a 6B-token Supervised Fine-Tuning (SFT) run that fixed a critical End-of-Sequence (EOS) token bug from earlier versions.

Despite its compact size, SmallCoder achieves state-of-the-art (SOTA) coding performance for <500M models, rivaling 1B–7B parameter LLMs.

Trained with support from Google’s TPU Research Cloud (TRC) program.

🚀 Key Results

⚖️ SmallCoder nearly matches Mistral 7B on HumanEval while being 23× smaller.

🧬 Model Architecture

A LLaMA-type causal decoder with standard Multi-Head Attention (MHA).

LlamaConfig(
  vocab_size=49152,               # StarCoder tokenizer
  hidden_size=768,
  num_hidden_layers=24,
  num_attention_heads=8,
  num_key_value_heads=8,
  intermediate_size=3072,
  max_position_embeddings=1024,
)

📚 Training Curriculum (4 Stages, 29.8B tokens)

🧩 Total ≈ 29.8 B tokens of curated curriculum learning.

📊 Detailed Benchmarks (Stage 4.1 SFT)

humaneval/mbpp were computed with manual evaluation (max_new_tokens=512, temp=0.2) due to SFT format truncation issues in lm-eval.

⚠️ Known Limitations

1. 1.Code-Specialized Model Tuned for Python and algorithmic reasoning. Poor performance on general text, math, and commonsense tasks.

1. 1.Short Context Trained on 1 024-token sequences only. Performance degrades on longer inputs.

1. 1.Tokenizer Bias Uses bigcode/starcoder BPE vocabulary — optimized for code, not prose.

💻 Usage Example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Beebey/smallcoder-303m"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).to(device)

prompt = """User: Write a Python function to compute Fibonacci numbers.
Assistant:"""
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

💡 Trained using the “User:” / “Assistant:” dialogue format.

🧾 Citation

If you use SmallCoder (303M) in your research, please cite:

@misc{smallcoder303m,
  title  = {SmallCoder: A 303M-parameter Code LLM trained from scratch},
  author = {Da Silva, Ilan},
  year   = {2025},
  url    = {https://huggingface.co/Beebey/smallcoder-303m},
  note   = {Trained with Google TPU Research Cloud (TRC) support}
}

🙏 Acknowledgements

This model was trained with support from the Google TPU Research Cloud (TRC) program. Special thanks to the open datasets that enabled this work: FineWeb, StarCoderData, Nemotron, and OpenWebMath.

🧩 Summary

🔬 SmallCoder (303M) demonstrates that a carefully designed <500M model can achieve near-SOTA coding performance, matching 1B-class models on HumanEval — proving that efficient, compact, open models still matter.