CoolFace
Modelpublic

Banaxi-Tech/slmoe-test

sourceHugging Faceupdated 2mo agoView on Hugging Face
0likes63downloads
Model Card

SLMoE Test - 100%

This model has been released as https://huggingface.co/BananaMind/BananaMind-2-SLMoE

This is the 100% checkpoint of an experimental sequence-routed base language model. It is not instruction tuned.

Architecture

FieldValue
Total parameters25,449,472
Active parameters per sequence7,902,208
Layers / hidden size8 / 256
Query / KV heads8 / 2
Head dimension32
Expert paths64 total / 13 active
Expert intermediate size56
Context4,096
Vocabulary8,192, tied

The router reads only the first 32 valid tokens and makes one global top-13 decision. Those expert identities are reused in every layer and stored in the KV cache for the complete generated response. Routing does not change per token. During pretraining, one packed 4,096-token sequence is the routing unit, and loss on its routing prefix is masked to prevent future-token leakage.

Training

FieldValue
Progress100%
Tokens seen59,999,518,720
Target tokens60,000,000,000
Hardware4 x NVIDIA H200
OptimizerFused AdamW
Peak LR0.003
Betas0.9, 0.95
Weight decay0.1
Precisionbfloat16 autocast
Token rangeFineWeb-HQFineWeb-EduDCLMCosmopedia v2FineMathNPSet-2
0.00B-12.00B35%30%20%10%4%1%
12.00B-24.00B32%30%18%11%7%2%
24.00B-39.00B28%29%16%13%11%3%
39.00B-51.00B24%27%13%14%18%4%
51.00B-60.00B20%24%9%15%26%6%

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Banaxi-Tech/slmoe-test"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)