Anurich/Jeeves-Small-95M

Jeeves (96M) — Looped Transformer

A compact instruction-tuned language model using Looped Transformer + Value Residual Learning. Trained with ChatML format for conversational AI and tool-calling capabilities.

Most compute-efficient model in its weight class — trained on only ~2B tokens, outperforms models trained on 20–150x more data.

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Anurich/Jeeves-Small-95M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Anurich/Jeeves-Small-95M", trust_remote_code=True)
model.eval()

# Use ChatML format (recommended for best results)
prompt = "<|im_start|>user\nWhat is photosynthesis?<|im_end|>\n<|im_start|>assistant\n"

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Note: trust_remote_code=True is required.

Chat Format (ChatML)

This model was fine-tuned using ChatML format. For best results, structure prompts like:

<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

Multi-turn Conversation

conversation = """<|im_start|>user
What is the speed of light?<|im_end|>
<|im_start|>assistant
The speed of light is approximately 299,792 kilometers per second.<|im_end|>
<|im_start|>user
How long does it take light to reach Earth from the Sun?<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(conversation, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Example Outputs

Benchmark Comparison

Zero-Shot Performance vs All Sub-200M Models

Models Jeeves Outperforms (with fewer parameters & less data)

vs Cerebras-GPT 111M (17% more params, similar data budget):

• —Jeeves wins on ALL shared benchmarks: HellaSwag +6.7pp, ARC-Challenge +10.2pp, PIQA +5.4pp, WinoGrande +3.6pp

vs OPT-125M (32% more params, 90x more training data):

• —Jeeves wins: HellaSwag +4.3pp, ARC-Challenge +3.9pp, PIQA +2.8pp, WinoGrande +0.8pp, GSM8K +1.5pp

vs GPT-Neo 125M (32% more params, 150x more training data):

• —Jeeves wins: HellaSwag +3.2pp, WinoGrande +0.6pp, GSM8K +1.4pp

vs GPT-2 137M (44% more params, 20x more training data):

• —Jeeves wins: HellaSwag +2.0pp, WinoGrande +2.0pp, GSM8K +1.0pp

vs Pythia 160M (68% more params, 150x more training data):

• —Jeeves wins on ALL shared benchmarks: HellaSwag +4.2pp, ARC-Challenge +8.7pp, PIQA +2.1pp, WinoGrande +0.5pp

Models That Beat Jeeves

SmolLM-135M and SmolLM2-135M outperform Jeeves on HellaSwag, PIQA, and MMLU — but were trained on 600B and 2T tokens respectively (300–1000x more data) using 64 H100 GPUs. Jeeves was trained on ~2B tokens.

Training Efficiency

Jeeves achieves 100–800x better benchmark-per-token efficiency than comparable models, demonstrating that architecture innovation (looped transformers + value residual learning) can dramatically reduce the data and compute needed to reach competitive performance.

Architecture

Jeeves uses a Looped Transformer — a single middle block is run multiple times with input injection, giving effective depth much larger than the unique parameter count.

Input → [Early Layers 0-10] → [Loop Block 11 × 6 iters] → [Late Layers 12-21] → Output
                                      ↑          |
                                      +----------+  (input injection)

Each loop iteration reuses the same weights, so the model gets 27 effective layers of processing with only 22 unique layer parameter sets.

Key Innovations

• —Looped Transformer (arXiv 2311.12424) — weight sharing via block looping for parameter efficiency
• —Value Residual Learning (arXiv 2410.17897) — first-layer value residuals prevent representation collapse in deep/looped networks
• —Input Injection — adds pre-loop hidden state back during each loop iteration for training stability
• —Grouped Query Attention — 9 query heads with 3 key-value heads for efficient inference

Training Pipeline

Special Tokens

Limitations

• —96M parameters — this is a small research model, not a production system
• —SmolLM/SmolLM2 (135M) achieve higher absolute scores with 300–1000x more training data
• —May hallucinate facts, especially for complex math or rare knowledge
• —Repetition in longer outputs is common at this scale
• —Best suited for simple Q&A, short-form generation, and research into efficient architectures

License

Apache 2.0

Citation

@misc{jeeves2026,
  title={Jeeves: Efficient Language Modeling with Looped Transformers and Value Residual Learning},
  author={Anurich},
  year={2026},
  url={https://huggingface.co/Anurich/Jeeves-Small-95M}
}