CoolFace
Modelpublic

FlameF0X/TinyMoE-100m-2x8-retrained

sourceHugging Facemitupdated 3mo agoView on Hugging Face
3likes361downloads
Model Card

TinyMoE-100M-2x8

TinyMoE-100M-2x8 is a compact, highly efficient Sparse Mixture of Experts (MoE) language model built upon the Mixtral/Mistral architecture. Designed for research, edge applications, and resource-constrained environments, this model leverages an expert-routing mechanism to balance a larger total parameter capacity with ultra-low computational overhead during inference.

[!IMPORTANT] On 05 July 2026 i update the weights with new ones. Pleace change the old weights with the new ones.

Model Details

  • —Architecture: Sparse Mixture of Experts (MoE)
  • —Total Parameters: 99,809,280 (~100M total parameters)
  • —Active Parameters per Token: 22,544,640 (~22.5M active parameters)
  • —Expert Configuration: 8 total local experts, 2 active experts routed per token (num_experts_per_tok": 2)
  • —Context Length: 1024 tokens
  • —Base Architecture: Mixtral / Mistral For Causal LM
  • —License: MIT

Parameter Breakdown

Unlike a standard dense model, an MoE model stores a larger footprint of parameters on disk but selectively activates only a subset for any given token during a forward pass:

ComponentTotal ParametersStatus During Inference
Embeddings (Input + LM Head)24,576,000Always Active
Attention Blocks (10 Layers)4,423,680Always Active
MoE Routers (10 Layers)30,720Always Active
Experts (8 Total across 10 Layers)70,778,8802 of 8 Active per Layer (~17.6M active)
Overall Footprint99,809,28022,544,640 Active per Token

Training Data

This model was trained on a HuggingFaceTB/smollm-corpus subsets cosmopedia-v2 and fineweb-edu-dedup

image image image image

Quick Start

You can load and experiment with this model using the Hugging Face transformers library:

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "FlameF0X/TinyMoE-100m-2x8-retrained"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

input_text = "Wikipedia is a free"
inputs = tokenizer(input_text, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
> Wikipedia is a free, open source of information about the world. It is a great resource for anyone who has been able to read and write in a way that is easy to read.
The first thing that is in the world of the internet is that it is not