Girinath11/recursive-language-model-198m
Mixture of Recursion Language Model — 198M (Adaptive Computation)
198M parameter language model Built from scratch on a single T4 GPU — no pretrained base, no fine-tuning. Novel self-supervised perplexity-guided adaptive computation via Mixture of Recursion (MoR).
Novel Architecture: Mixture of Recursion (MoR)
Traditional transformers apply the same depth to every input — simple and complex sentences both go through all N layers. This wastes compute on easy inputs and under-processes hard ones.
MoR fixes this with self-supervised perplexity-guided routing:
High PPL (>50) → Model struggling → 5 recursive steps
Mid PPL (20–50) → Uncertain → 3 recursive steps
Low PPL (<20) → Model confident → 1 recursive stepNo manual labels needed — the router learns difficulty directly from the model's own perplexity signal during training.
Key Components
NaN-Safe FP16 Training
Standard -inf masking causes NaN during fp16 mixed-precision training. MoR uses -1e4 masking + pre-softmax clamping — completely stable:
scores = scores.clamp(min=-1e4, max=1e4)
attn = F.softmax(scores, dim=-1)
attn = torch.nan_to_num(attn, nan=0.0)Result: 0 NaN batches across all 150K training steps.
Performance — Honest Evaluation
Training Val Set (in-distribution)
Evaluated on 2,000 held-out samples from the same conversational distribution as training data (HH-RLHF + UltraChat + Alpaca-GPT4):
Fresh Test Set (out-of-distribution)
Evaluated on 500 fresh HH-RLHF test samples (never seen during training). GPT-2 Medium evaluated on identical samples for fair comparison:
Honest note: MoR scores higher PPL on the fresh test set than GPT-2 Medium. The 15.37 val PPL was measured on in-distribution data — the model shows signs of overfitting to the training distribution. The architecture is novel and the routing mechanism works correctly, but the model needs more diverse training data and longer training to generalize better. The val PPL (15.37) should not be directly compared to GPT-2 Medium's standard benchmark PPL — they are measured on different datasets.
What the Results Actually Show
Architecture works — training is stable, loss decreases
Router works — 0 NaN batches, routing signals valid
Novel contribution — perplexity-based self-supervised routing
Generalization gap — val PPL 15.37 vs test PPL 43.21
More data needed — 150K samples is small for this model sizeQuick Start
pip install transformers torchfrom transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"Girinath11/recursive-language-model-198m",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
"Girinath11/recursive-language-model-198m",
trust_remote_code=True
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()Chat Format (required)
The model was trained with a specific chat format — always use this:
def chat(question, max_new_tokens=150, temperature=0.7, top_p=0.9):
prompt = f"<|user|>\n{question}\n<|assistant|>\n"
inputs = tokenizer(
prompt,
return_tensors="pt",
add_special_tokens=False
).to(device)
with torch.no_grad():
outputs = model.generate(
inputs["input_ids"],
max_new_tokens = max_new_tokens,
temperature = temperature,
top_p = top_p,
do_sample = True,
)
full = tokenizer.decode(outputs[0], skip_special_tokens=True)
return full.split("<|assistant|>")[-1].strip() \
if "<|assistant|>" in full else full
print(chat("What is machine learning?"))
print(chat("Explain neural networks simply"))Training Details
Dataset — 150K Conversational Samples
Training Config
GPU : NVIDIA Tesla T4 (15.6 GB VRAM)
Platform : Kaggle (single GPU)
Epochs : 2
Total steps : 150,000
Training time : ~9h 12m
Batch size : 2
Grad accum : 32 (effective batch = 64)
Max seq len : 512
Learning rate : 1e-4
LR schedule : Linear warmup + Cosine decay
Warmup steps : 500
Optimizer : AdamW (β=0.9, 0.95, ε=1e-8)
Weight decay : 0.01
Grad clip : 1.0
Mixed precision: FP16 (AMP)
Loss:
Total = LM loss + 0.1 × Router lossTechnical Innovation
Self-Supervised Perplexity Routing
# During training — no manual labels needed!
sample_ppl = exp(per_sample_ce_loss)
if sample_ppl < 20: pseudo_label = 0 # simple
elif sample_ppl < 50: pseudo_label = 1 # medium
else: pseudo_label = 2 # complex
router_loss = CrossEntropyLoss(router_logits, pseudo_label)
total_loss = lm_loss + 0.1 * router_lossWhy it works:
- Router learns difficulty FROM the model's own performance
- As training progresses, more samples become "simple" — natural curriculum
- No annotation cost, no bias from human difficulty ratings
What I Learned / What to Improve Next
This model is my first from-scratch LLM — I learned a lot:
What worked:
NaN-safe attention masking (-1e4)
Self-supervised routing signal
Stable FP16 training (0 NaN batches)
Perplexity drops meaningfully across epochs
What needs improvement:
More diverse training data (150K is small)
Longer training (2 epochs not enough)
Stronger regularization to prevent overfitting
Better evaluation on neutral benchmarks from day 1Limitations
- Context window: 512 tokens max
- English only: Not suitable for other languages
- Small training set: 150K samples — commercial models use 100B+
- Generalization: Shows overfitting to training distribution
- Repetition: May loop in very long generations (>200 tokens)
- Factual accuracy: May hallucinate — do not use for facts
Intended Use
Citation
@misc{girinath2026mor,
author = {Girinath V},
title = {Mixture of Recursion: Self-Supervised Perplexity-Guided
Adaptive Computation for Language Models},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Girinath11/recursive-language-model-198m}},
note = {198M parameter LM trained from scratch with adaptive
recursive computation via perplexity-based routing}
}Acknowledgments
- Anthropic — HH-RLHF dataset
- Tsinghua University — UltraChat dataset
- Vicgalle — Alpaca-GPT4 dataset
- HuggingFace — Transformers library
- Kaggle — Free GPU access
Model status : Research / Educational use Last updated : March 2026 License : MIT