RtaForge/Anvaya-Rabbit-2.7B

1---2language:3  - en4license: other5license_name: anvaya-model-license6license_link: LICENSE.md7library_name: transformers8pipeline_tag: text-generation9tags:10  - state-space-model11  - rtassm12  - rabbit13  - custom-architecture14  - causal-lm15  - tool-use16  - frontier17  - research18  - anvaya19  - rtaforge20---21 22# Anvaya-Rabbit-2.7B23 24**RtaForge · Frontier AI Research Lab**  25*ANVAYA — post-transformer state-space research*26 27<img src="rabbit_slm.png" width="220" alt="Anvaya Rabbit"/>28 29**A 2.7B research checkpoint for tool-use on a custom state-space architecture.**30 31Rabbit is the tool-calling archetype (“Dagger”) in RtaForge’s ANVAYA program: models trained on **RtaSSM / Tungsten**, a non-transformer architecture, with a Rust-native training stack. The research question is whether SSMs can carry **fast, inspectable tool-use** without inheriting the transformer stack. This page ships **0.72TG-beta**.32 33> Research beta — honestly scoped, not a finished production model. Deeper reasoning is a separate architecture track.34 35---36 37## Model highlights38 39| | |40| --- | --- |41| **Lab** | RtaForge — Frontier AI Research Lab |42| **Program** | ANVAYA |43| **Architecture** | RtaSSM v7.4 (“Tungsten”) — state-space LM |44| **Parameters** | 2.7B |45| **Layers / width** | 64 × 2560 |46| **Tokenizer** | EleutherAI GPT-NeoX (50,280 vocab) |47| **Context** | 4K trained & verified · 8K in active engineering |48| **Role in the lab** | Fast tool-calling probe (not a deep reasoner) |49| **Training stack** | Rust-native · Gurukul curriculum · Fortress/CLAIM3 |50| **License** | [Anvaya Model License](LICENSE.md) — free for personal/research use; org use is revenue-gated |51 52**What this release is.** An artifact from ongoing frontier work on SSM tool-use — architecture, curriculum, and forgetting protection included, not a LoRA on someone else’s base.53 54**What this release is not.** A general reasoner or a production agent runtime. Math/logic depth continues on a purpose-built track; Rabbit stays the tool-use lane.55 56---57 58## Quick start59 60Requires the RtaForge runtime (loads the custom SSM implementation behind the Transformers wrapper):61 62```bash63pip install rtaforge transformers64```65 66```python67import torch68from transformers import AutoTokenizer, AutoModelForCausalLM69 70model_id = "RtaForge/Anvaya-Rabbit-2.7B"71# Recommended checkpoint for this release:72#   base/Anavya-Rabbit-2.7B-0.72TG-beta.safetensors   (or .pt — byte-identical)73# (Historical spelling "Anavya" is retained in filenames.)74 75tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")76model = AutoModelForCausalLM.from_pretrained(77    model_id,78    trust_remote_code=True,79    torch_dtype=torch.bfloat16,80).eval()81 82# If your install expects an explicit weight path:83# model = ...  # see rtaforge docs for checkpoint loading helpers84 85prompt = "List three tools you would call to answer a weather question, then stop."86inputs = tokenizer(prompt, return_tensors="pt")87with torch.no_grad():88    out = model.generate(89        **inputs,90        max_new_tokens=128,91        do_sample=True,92        temperature=0.7,93        top_p=0.9,94        repetition_penalty=1.1,95    )96print(tokenizer.decode(out[0], skip_special_tokens=True))97```98 99> **Tip:** constrain `max_new_tokens` and/or detect repetition loops in unconstrained generation — see [Known limits](#known-limits).100 101---102 103## Architecture104 105Rabbit belongs to the **RtaSSM** family — state-space language models under ANVAYA, RtaForge’s post-transformer research program.106 107| Component | Role in this release |108| --- | --- |109| **Tungsten (RtaSSM v7.4)** | Production training architecture for this lineage |110| **Gurukul** | Multi-phase curriculum (logic → math → unified bridge SFT) |111| **Fortress / CLAIM3** | Engram-based forgetting protection on later phases |112| **Constitutional validation** | Training-time checks in the Rust stack |113 1140.71TG-beta was a **unified bridge SFT** pass (tool-calling + polish + STEM, conversational / masked-loss training on top of the 0.7TG math checkpoint) that closed a general-capability regression from the preceding math-focused phase while keeping the math floor intact.115 116**0.72TG-beta** adds one further, targeted pass: a dedicated math-instruct SFT on top of the 0.71TG bridge. Fortress/CLAIM3 protected the bridge (tool-call / general) manifold, so general benchmarks held within noise while GSM8K reached the best value on the lineage. The 0.71TG bridge remains the last *broad* SFT pass; deeper reasoning work continues on a separate architecture.117 118---119 120## Evaluation121 122Evaluated with [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness).  123ARC-Easy / ARC-Challenge / HellaSwag / WinoGrande: **0-shot**. GSM8K: **5-shot**.  124ARC / HellaSwag report **normalized accuracy** (`acc_norm`); WinoGrande reports **accuracy** (`acc`); GSM8K reports **exact match (flexible)**.125 126### 0.72TG-beta vs 0.71TG-beta (Phase 4 math-instruct)127 128| Task | 0.71TG-beta | **0.72TG-beta** |129| --- | ---: | ---: |130| ARC-Easy (`acc_norm`) | 0.5960 | **0.5905** |131| ARC-Challenge (`acc_norm`) | 0.3746 | **0.3669** |132| HellaSwag (`acc_norm`) | 0.6286 | **0.6313** |133| WinoGrande (`acc`) | 0.6298 | **0.6346** |134| GSM8K 5-shot (strict / flex) | 0.0713 / 0.0720 | **0.0857 / 0.0857** |135| Tool-call loop rate | 16.9% | **9.1%** |136| GSM8K free-gen loop rate | 7.8% | 10.6% |137 138**0.72TG** is the current release. General benchmarks vs 0.71TG sit inside noise; GSM8K is the best value on the lineage; tool-call loop rate roughly halves. GSM8K free-generation loop rate ticks up (7.8% → 10.6%) — a real trade on unconstrained math gen.139 140### Scores by checkpoint141 142| Task | 0.5TG | 0.6TG | 0.7TG | 0.71TG | **0.72TG** | 1.0 |143| --- | ---: | ---: | ---: | ---: | ---: | ---: |144| ARC Easy | 0.5833 | 0.6174 | 0.5741 | 0.5960 | **0.5905** | — |145| ARC Challenge | 0.3703 | 0.3831 | 0.3660 | 0.3746 | **0.3669** | — |146| HellaSwag | 0.6063 | 0.6485 | 0.6228 | 0.6286 | **0.6313** | — |147| WinoGrande | 0.6164 | 0.6259 | 0.6417 | 0.6298 | **0.6346** | — |148| GSM8K (5-shot, flex) | — | — | 0.0705 | 0.0720 | **0.0857** | — |149 150**1.0** is reserved for the next public milestone.151 152**How to read this.** 0.6TG (GAPA Phase 1 Logic final, `pearl_step_122032`) is the local peak on ARC-Easy / ARC-Challenge / HellaSwag. The Math phase (0.7TG) traded some of that general-benchmark performance for GSM8K capability — the 0.6→0.7 dips on ARC are ~1–2σ (stderr typically ±0.010–0.014 on ARC/WinoGrande, ±0.005 on HellaSwag): real but modest. The 0.71TG bridge partially recovers ARC while keeping the math floor. 0.72TG's math-instruct pass then moves GSM8K to its best lineage value with ARC / HellaSwag / WinoGrande deltas vs 0.71TG inside 1σ.153 154GSM8K is blank for 0.5TG and 0.6TG because those suites did not include it — it was only added when the Math phase landed. Not missing data; not measured.155 156**Provenance.** 0.5TG: `rabbit_dagger_tungsten_2026-07-06` (0.5-alpha-base / Tungsten baseline eval). 0.6TG: `eval_results/rabbit_dagger_phase1_final_2026-07-23/` — lm-eval-harness, HF backend, bf16, converted non-strict to `Mamba2ForCausalLM` (missing=0, unexpected=0), L4. 0.7TG / 0.71TG / 0.72TG: same harness conventions as published on this card.157 158### Bridge run detail (0.7TG → midpoint → 0.71TG)159 160| Task | 0.7TG | bridge @10k | **0.71TG-beta** |161| --- | ---: | ---: | ---: |162| ARC Easy | 0.5741 | 0.5888 | **0.5960** |163| ARC Challenge | 0.3660 | 0.3712 | **0.3746** |164| HellaSwag | 0.6228 | 0.6296 | **0.6286** |165| WinoGrande | 0.6417 | 0.6298 | **0.6298** |166| GSM8K (5-shot) | 0.0705 | 0.0804 | **0.0720** |167 168We do **not** train against these benchmarks or select checkpoints by them. They are smoke-test signals for the lineage, not the product target (tool use).169 170---171 172## Known limits173 174- **Math ceiling ~8–9% GSM8K** on this lineage — 0.72TG raised GSM8K to the best value here (0.0857) but did not break the ceiling. Failures are real reasoning errors, not formatting mismatches; deep multi-step math is not this architecture's lane.175- **WinoGrande** took a one-time ~1.2pt cost from the tool-call / polish mix; it did not degrade further with more bridge / instruct training.176- **Repetition loops** — tool-call loop rate **16.9% → 9.1%** on 0.72TG; GSM8K free-gen loop rate **7.8% → 10.6%**. Constrain `max_new_tokens` and/or detect repeats in deployment.177 178---179 180## Checkpoints181 182Supported baseline for continued work:183 184```text185base/Anavya-Rabbit-2.7B-0.72TG-beta.safetensors  ← current (0.72TG-beta)186base/Anavya-Rabbit-2.7B-0.72TG-beta.pt            ← byte-identical to the .safetensors187base/Anavya-Rabbit-2.7B-0.71TG-beta.pt188base/Anavya-Rabbit-2.7B-0.7TG-beta.pt189base/Anavya-Rabbit-2.7B-0.6TG-beta.pt190base/Anavya-Rabbit-2.7B-0.5TG-beta.pt191```192 193All base checkpoints are safetensors-format weights; earlier releases carry a `.pt` extension for continuity with prior tooling. 0.72TG-beta is the first to ship both extensions (identical bytes) — prefer `.safetensors`.194 195Earlier prototypes under `deprecated/` are kept for historical reference and are **not** recommended as training starts.196 197| Capability | Status |198| --- | --- |199| Training | ✅ |200| Continued training | ✅ |201| 4K context training | ✅ Verified |202| 8K context training | Active engineering |203 204---205 206## Roadmap207 208| Release | Focus |209| --- | --- |210| 0.5TG-beta | Tungsten baseline |211| 0.6TG-beta | Logic phase |212| 0.7TG-beta | Mathematics phase |213| 0.71TG-beta | Unified bridge SFT (tool-calling + polish + STEM) |214| **0.72TG-beta** | **Dedicated math-instruct pass — best-lineage GSM8K, general capability held** |215| 1.0 | Instruction / tool-call polish, expanded benchmarks |216 217Rabbit stays the **fast tool-use** lane. Reasoning-depth work continues on a separate architecture.218 219---220 221## License222 223Released under the [Anvaya Model License Agreement](LICENSE.md). In brief:224 225- **Free** for an individual’s own personal use or independent research.226- **Organizational use** (by or for any company, non-profit, or other entity, regardless of revenue) requires either staying under the revenue threshold (INR 1 crore / USD 100,000 per year, whichever is lower) or an Enterprise License.227- **Resale, paid bundling, or hosted API access** to the model or a derivative always requires RtaForge’s prior written permission, regardless of revenue.228- **Derivatives and fine-tunes** must credit both Anvaya-Rabbit-2.7B and RtaForge OPC Private Limited by name.229- **No content safeguards are mandated** — you are solely responsible for what you generate and how you deploy it.230 231The full [LICENSE.md](LICENSE.md) is binding; this summary is for convenience only.232 233---234 235## About RtaForge236 237**RtaForge** is a Frontier AI Research Lab. We invent and train post-transformer **state-space** systems end-to-end — architectures, training protocols, inference runtimes, and models — rather than fine-tuning imported transformer bases.238 239**ANVAYA** is the research program behind this release: RtaSSM architectures (Tungsten and siblings), Rust-native training (Anvil / Gurukul / Fortress), weight migration (Subsuminator), and a model family (Rabbit → tool-use; Raccoon and others → reasoning and beyond). Related public work includes structural heists and methodology notes such as [FORGEry](https://doi.org/10.5281/zenodo.20374967).240 241Rabbit is one artifact from that program — evidence, not the whole lab.242 243- Org: [huggingface.co/RtaForge](https://huggingface.co/RtaForge)244- Heists: [github.com/Rta-Forge/heists-galore](https://github.com/Rta-Forge/heists-galore)245- Contact: guha@rtaforge.in246 247© RtaForge OPC Private Limited248