Prannesshkva/ISOM-R1-Edge-130M-MoE-Beta-Prototype

ISOM-R1-Edge-130M-MoE: Continuous Recurrent SSM + Mixture-of-Experts

0.0469 MB Invariant State Footprint • Verified on NVIDIA Tesla T4 Cloud GPU • 100K Continuous Token Stream

<p align="center"> <a href="https://doi.org/10.5281/zenodo.22649142"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.22649142.svg" alt="DOI"></a> <a href="https://www.linkedin.com/in/prannesshkva/"><img src="https://img.shields.io/badge/LinkedIn-PranneshK.V.A.-blue?logo=linkedin" alt="LinkedIn"></a> <img src="https://img.shields.io/badge/Architecture-ContinuousSSM%2BMoE-yellow.svg" alt="Architecture"> <img src="https://img.shields.io/badge/Params-134.89MTotal%2F58.27MActive-orange.svg" alt="Params"> <img src="https://img.shields.io/badge/StateFootprint-0.0469MBConstant-brightgreen.svg" alt="State"> <img src="https://img.shields.io/badge/TeslaT4-HardwareAudited-brightgreen.svg" alt="Tesla T4"> <a href="https://huggingface.co/spaces/Prannesshkva/ISOM-Benchmark"><img src="https://img.shields.io/badge/HFSpace-Benchmark_Suite-yellow.svg" alt="Space"></a> </p>

Overview

ISOM-R1-Edge-130M-MoE is an ultra-compact 134.89-million parameter (58.27M active per token) continuous recurrent state-space architecture. Engineered specifically as a high-speed speculative decoding drafter and edge deliberation engine, it processes arbitrarily long token streams without materializing quadratic attention matrices.

📊 Audited Empirical Hardware Telemetry (NVIDIA Tesla T4, Kaggle Cloud)

Evaluated on an NVIDIA Tesla T4 (14.56 GB / 14,911.7 MB total VRAM, PyTorch 2.10.0+cu128, CUDA 12.8, Kaggle Cloud) across an authentic, unpadded continuous literature stream (Pride and Prejudice, 728,846 characters) from 10,000 up to 100,000 continuous tokens:

Continuous Recurrent Scaling (10,000 to 100,000 Tokens)

Key Architectural Verification: Across the entire 100,000 continuous token stream, the recurrent hidden state footprint remains strictly invariant at 0.0469 MB (48 KB). The continuous Cayley SO(d) manifold preserves numerical isometry, delivering processing speeds scaling up to 21,049.0 tokens/second on a single Tesla T4 GPU.

Model Specifications

Quickstart Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Prannesshkva/ISOM-R1-Edge-130M-MoE-Beta-Prototype"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

prompt = "Solve step by step: If a car travels 90 km/h for 3.5 hours, what is the total distance traveled?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        temperature=0.7,
        do_sample=True
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Optional: Hybrid Neuro-Symbolic Agent Pipeline
from modeling_isom_130m_moe import ISOMDeliberativeAgent
agent = ISOMDeliberativeAgent(model=model, tokenizer=tokenizer)
result = agent.solve("Solve step by step: A train travels 60 km in 45 minutes. What is its speed in km/h?")
print("Agent Trace:", result.get("thought_trace"))
print("Final Answer:", result.get("final_answer"))

⚠️ Training Notice: MoE Load-Balancing

ISOMStaticMoE uses softmax top-2 routing with weight renormalization but does not include an auxiliary load-balancing loss. During inference this has no effect — the model routes normally. During fine-tuning, the absence of a balancing term means the router is free to concentrate all tokens on a small subset of experts (expert collapse), leaving the remaining experts with near-zero gradient signal and wasted capacity.

Mitigation for fine-tuning: Add a Switch Transformer-style auxiliary loss to your training loop before calling loss.backward():

# After computing CE loss, add auxiliary load-balancing loss
router_logits = ...  # collect router_logits from each ISOMStaticMoE layer
probs = torch.softmax(router_logits, dim=-1)                        # [T, E]
f_i = probs.mean(dim=0)                                             # expert fraction
P_i = (probs > 0).float().mean(dim=0)                              # expert dispatch rate
aux_loss = config.num_experts * (f_i * P_i).sum()
total_loss = ce_loss + 0.01 * aux_loss

Without this, or an equivalent expert-parallelism regularizer, training runs longer than ~1,000 steps risk expert utilization collapse to 2 of 8 experts.

Citation & Licensing

@software{isom_edge_130m_2026,
  author = {Prannessh K.V.A.},
  title = {ISOM-R1-Edge-130M-MoE: Continuous Recurrent SSM + Mixture-of-Experts Drafter},
  year = {2026},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.22649142},
  url = {https://doi.org/10.5281/zenodo.22649142}
}

• —Sole Author & Architect: Prannessh K.V.A.
• —LinkedIn: Prannessh K.V.A.
• —License: Governed by CC BY-NC-ND 4.0 (Non-Commercial Research) & Enterprise Commercial Terms. See LICENSE.

Notice of Non-Endorsement & Independent Lineage

[!IMPORTANT] Independent Architecture Work: ISOM-R1-Edge-130M-MoE is an original standalone research architecture engineered solely by Prannessh K.V.A. (Author, Architect &amp; IP Holder). It implements continuous isometric state operator manifolds (Cayley SO(d)) combined with 8 SwiGLU Mixture-of-Experts feedforward layers. Governed by CC BY-NC-ND 4.0 & Enterprise Commercial Terms (see LICENSE).