kaptaan45/KaptaanLM-Vanguard-25M

KaptaanLM-Vanguard-25M: Master Fleet Repository

KaptaanLM-Vanguard is the flagship 25.18M parameter model family of the KaptaanLM project. Pre-trained on 5.636 Billion tokens across 8 curated corpora and aligned via the Universal Scratchpad Contract with Schema-Aligned Direct Preference Optimization (DPO), Vanguard sets a new standard for reasoning density under 50M parameters.

• —GitHub Repository: https://github.com/rudy-07/KaptaanLM
• —Evaluation Protocol: EleutherAI lm-evaluation-harness (11,858 official academic questions)
• —Architecture: 12 layers, dmodel=384, 12 attention heads, 4 KV heads (3:1 GQA), dffn=1024 (SwiGLU), 16,384 Tied BPE vocabulary
• —Context Length: 2,048 tokens (512 sliding window attention)
• —Total Parameters: 25,176,192 (25.18M)
• —Non-Embedding Parameters: 18,884,736 (18.88M / 75.01% reasoning capacity)

Model Fleet Directory (Checkpoints Available in this Repository)

Official Academic Benchmark Results (11,858 Questions)

Evaluated under the EleutherAI benchmark protocol under Regime 1 (Deterministic Anti-Repetition Greedy: temp=0.0, rep_penalty=1.12):

Key Scientific Insights

1. 1.Mathematical Reasoning Breakthrough: Vanguard-Omni-DPO solved 46 GSM8K problems (3.49%), representing 2.3x more problems solved than SmolLM-135M (20 problems / 1.52%) at 1/5th the model size.
2. 2.DPO Structural Regularization: Schema-Aligned DPO prevented conversational format lock-in on continuous prose (restoring LAMBADA from 0.00% to 13.39%) and pruned discursive loops (elevating GSM8K from 0.08% to 3.49%, a 43.6x increase).

Usage & Python Quickstart

import torch
from tokenizers import Tokenizer
from kaptaan.config import CONFIG_25M
from kaptaan.torch.model import KaptaanForCausalLM

tokenizer = Tokenizer.from_file("tokenizer.json")
config = CONFIG_25M
model = KaptaanForCausalLM(config)
model.load_state_dict(torch.load("KaptaanLM-Vanguard-Omni-DPO-1.0.pt", map_location="cpu"))
model.eval()

prompt = "<|im_start|>user\nWrite a Python function to check if a number is prime.\n<|im_end|>\n<|im_start|>assistant\n"
tokens = tokenizer.encode(prompt).ids
input_ids = torch.tensor([tokens], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(
        input_ids,
        max_new_tokens=256,
        temperature=0.0,
        repetition_penalty=1.12,
        stop_token_ids=[0, 2]
    )

print(tokenizer.decode(output_ids[0].tolist()))

Citation

@software{kaptaanlm2026,
  author = {Shekhar, Rudransh},
  title = {KaptaanLM: Conversational and Reasoning Small Language Models (10M - 100M -> MoE)},
  url = {https://github.com/rudy-07/KaptaanLM},
  year = {2026}
}