nuroai/Avalon-2B
322
<div align="center">
AVALON-2B
Adaptive Vision-Augmented Language ON-device
The First Sub-3B Self-Reflective Language Model
   
Paper | GitHub | GGUF Version
</div>
Model Description
AVALON-2B is the first sub-3B parameter language model to implement Self-Reflective Retrieval-Augmented Generation (Self-RAG) with learned reflection tokens. Built upon Qwen 3.5 2B, AVALON introduces a novel training pipeline that teaches the model to generate retrieval decision tokens without external retrieval infrastructure.
Key Innovations
- First Sub-3B Self-RAG: Breaks the 7B parameter barrier for self-reflective capabilities
- On-Device Ready: 1.5GB quantized (Q4KM) runs at 40+ tok/s on Apple M3
- 82.5% Token Accuracy: Reliable generation of
[Retrieval],[No Retrieval], and[Utility:X]tokens - No Catastrophic Forgetting: +0.41% MMLU improvement over base model
Self-RAG Tokens
AVALON generates special reflection tokens to enable adaptive retrieval:
Benchmarks
vs Gemma 4 E2B (Head-to-Head)
On-Device Performance
Tested with Q4KM quantization (1.5GB):
Usage
Recommended System Prompt
For best performance, use this system prompt:
You are AVALON, a self-reflective AI assistant. Before answering any question:
1. Determine if you need external information by generating [Retrieval] or [No Retrieval]
2. For time-sensitive questions (news, current events, prices), always use [Retrieval]
3. For factual knowledge (capitals, math, definitions), use [No Retrieval]
4. End every response with [Utility:X] where X is 1-5 rating of response quality
Be concise and accurate. If you're uncertain, acknowledge it.Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("nuroai/Avalon-2B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("nuroai/Avalon-2B")
SYSTEM_PROMPT = """You are AVALON, a self-reflective AI assistant. Before answering any question:
1. Determine if you need external information by generating [Retrieval] or [No Retrieval]
2. For time-sensitive questions (news, current events, prices), always use [Retrieval]
3. For factual knowledge (capitals, math, definitions), use [No Retrieval]
4. End every response with [Utility:X] where X is 1-5 rating of response quality
Be concise and accurate. If you're uncertain, acknowledge it."""
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "Who won the 2024 US presidential election?"}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
# Output: [Retrieval]I need current information to answer this question...[Utility:4]Ollama (Recommended for Local Use)
# Download GGUF version
ollama pull nuroai/avalon-2b
# Run
ollama run avalon-2b "What is quantum computing?"
# Output: [No Retrieval]Quantum computing is a type of computation that...[Utility:5]llama.cpp
# Download Q4_K_M GGUF (1.5GB)
wget https://huggingface.co/nuroai/Avalon-2B-GGUF/resolve/main/avalon-2b-q4km.gguf
# Run inference
./llama-cli -m avalon-2b-q4km.gguf -p "What is the capital of Japan?" -n 128Training Details
Architecture
- Base Model: Qwen 3.5 2B (18 GDN + 6 Softmax attention layers)
- Parameters: 1.88B total
- Context Length: 32K tokens
- Vocabulary: 248K tokens (including Self-RAG special tokens)
Training Configuration
- Method: LoRA with
modules_to_save=["embed_tokens", "lm_head"] - Data: 201K synthetic samples (80% Self-RAG, 20% general)
- Hardware: 8x NVIDIA A100 80GB
- Duration: ~6 hours
- Learning Rate: 2e-5
- Epochs: 2
Critical Insight
The key to successful Self-RAG at sub-3B scale is including embedding layers in training:
peft_config = LoraConfig(
r=64,
lora_alpha=128,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
modules_to_save=["embed_tokens", "lm_head"], # CRITICAL for Self-RAG tokens
lora_dropout=0.05,
)Without modules_to_save, Self-RAG token accuracy drops from 82.5% to 12%.
Quantization
Limitations
- Mathematical Reasoning: GSM8K performance drops 4.5% due to token interruption in multi-step calculations
- English Only: Trained and evaluated exclusively on English data
- Static Retrieval Decision: Binary decision at generation start; cannot adapt mid-response
Citation
@article{ponnada2026avalon,
title={AVALON-2B: The First Sub-3B Self-Reflective Language Model for On-Device Deployment},
author={Ponnada, Akhil and Arvapalli, Naga Sri},
journal={arXiv preprint},
year={2026}
}License
Apache 2.0
Authors
- Akhil Ponnada - akhil@nuroailabs.com
- Naga Sri Arvapalli - nagasri3007@gmail.com
Contact
- Organization: Nuro AI Labs Limited
- GitHub: Nuro-Labs/avalon-2b