Modelpublic

nepetai/ivis-400m-gpu

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes925downloads

Model Card

Ivis-400M-Nepetai

Ivis Nepetai هو نموذج لغوي هجين يجمع ثلاث معماريات في كتلة واحدة: Transformer (GQA + RoPE) + Mamba SSM + Mixture of Experts.

A hybrid language model combining three architectures in a single stack: Grouped-Query Attention Transformer, Mamba State-Space layers, and sparse MoE.

Model details | تفاصيل النموذج

Property	Value
Total parameters	450,993,024 (~451M)
Active parameters	~250M (MoE: 4 experts × 2 active)
Layers	10
Hidden size	1,152
Attention	GQA — 18 query heads / 6 KV heads, head_dim 64, RoPE θ=10K
Mamba SSM	layers 3 & 7 · dstate=16 · dconv=4 · expand=2
MoE	layers 6–9 · 4 experts × 2 active · hidden 2,304
Context length	256 tokens
Vocabulary	65,536 (SentencePiece BPE)
Languages	العربية + English

What is it for? | ما تخصصه

—توليد نصوص ثنائية اللغة (عربي/إنجليزي)
—التحقق العملي من استقرار المعمارية الهجينة (Transformer+Mamba+MoE) أثناء التدريب
—نموذج بحثي تجريبي (research preview)

Training status | حالة التدريب

Trained on Kaggle Tesla P100-16GB (PyTorch 2.2.0+cu118), Adafactor lr=3e-4. This is an experimental pilot checkpoint from a short validation run (40 steps) — not a fully-trained production model. Loss decreased from ~2540 → ~57 early in the run.

تم التدريب على GPU مجاني (Kaggle P100) لعدة خطوات تحقق فقط — هذا ليس نموذجاً مدرّباً بالكامل.

Files | الملفات

File	Description
`ivis-400m-pytorch.pt`	Full PyTorch checkpoint (statedict + modelinfo)
`ivis-400m.gguf`	GGUF container (f16, 910MB, all 135 tensors)
`config.json`	Architecture hyperparameters
`tokenizer.model` / `tokenizer.vocab`	SentencePiece BPE tokenizer (AR+EN)
`infer_numpy.py`	Local inference engine — pure numpy, no PyTorch needed
`convert_to_gguf.py`	Torch-free .pt → GGUF converter

Run locally | التشغيل المحلي

bash

pip install gguf sentencepiece numpy   # no torch required!

python infer_numpy.py --gguf ivis-400m.gguf \
  --tokenizer tokenizer.model \
  --prompt "البرمجة هي" --tokens 40

# measured on CPU: d_model=1152 layers=10 heads=18/6 moe=4@L6+
# generation speed ≈ 7 tokens/s (numpy, single thread)

## Related | نماذج مرتبطة

- [`nepetai/ivis-50m-pilot`](https://huggingface.co/nepetai/ivis-50m-pilot) — same architecture family at 55M scale, **fully validated end-to-end** with measured training curve (loss 9.20→0.92 over 300 steps), GGUF conversion and a pure-numpy inference engine that runs without PyTorch.

## Citation

@misc{ivis-nepetai-400m, title={Ivis-400M-Nepetai: Hybrid Transformer-Mamba-MoE Language Model}, author={Nepetai Arabic}, year={2026}, url={https://huggingface.co/nepetai/ivis-400m-gpu} }