Sakatepon/Brujula-150M

Brújula-150M

A 153.6M-parameter decoder-only language model, trained from scratch on FineWeb-Edu as part of a solo, consumer-GPU hobby project. Brújula ("compass" in Spanish) is a deliberately minimal DeepSeek-style architecture: Multi-head Latent Attention (MLA) + RoPE + SquaredReLU FFN, with tied embeddings, trained with a hybrid Muon + AdamW optimizer.

This is a base completion model — it continues text; it is not instruction-tuned and not safety-tuned. It writes fluent, on-topic educational-web prose; it is not a reliable source of facts.

Results

Perplexity (lower is better), evaluated on a fixed local harness at context length 1024:

Honest reading: Brújula-150M beats GPT-2 small on FineWeb-Edu val (its home turf), but loses on WikiText-103 — a data-distribution effect (FineWeb-Edu is narrower than GPT-2's WebText), not a capacity problem. It's the main known limitation, stated openly, not hidden.

The Brújula family

Usage

Custom architecture code, so pass trust_remote_code=True:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Sakatepon/Brujula-150M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()

ids = tok("The mitochondria is the", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Tip: it's a small base model — use sampling and cued/definitional prompts ("X is the … ") rather than bare nouns. Greedy decoding tends to repetition-loop.

Architecture

Training

FineWeb-Edu (~5B tokens), 1 epoch (76,293 steps), hybrid Muon (matrix weights) + AdamW (embeddings/norms), peak LR 1.7e-3, batch 64, bf16, ~12.3h on a single cloud GPU (the 15M/18M siblings train locally on one Intel Arc B580).

The story

Brújula is the honest output of a part-time, single-consumer-GPU project. It started from an over-engineered 331M model (with a bolted-on "engram" retrieval mechanism + dropout + torch.compile + gradient checkpointing) that displayed low loss but evaluated ~6× worse in clean inference — a broken train/eval stack. The fix was ruthless minimalism: strip every unvalidated component, keep MLA + RoPE + SquaredReLU + tied embeddings, A/B each change honestly. A 15M minimal model then beat the 331M one by ~2.5×. Adding the Muon optimizer and scaling to 150M produced this. The journey is the point; the perplexity is just where it landed.

Limitations

• —Base completion model — not instruction-tuned, no safety tuning.
• —English only, educational-web distribution (FineWeb-Edu); weaker out-of-distribution (the WikiText gap).
• —Not a knowledge base — at 150M it produces plausible prose but unreliable facts.
• —Short context (1024); no KV-cache in this reference implementation.

License & attribution

• —Model + code: Apache-2.0.
• —Training data: FineWeb-Edu (ODC-BY).
• —Built on ideas from: DeepSeek-V2 (MLA), Muon optimizer, Primer (SquaredReLU), GPT-2 (BPE tokenizer).