OliverSundaram/MoE-Study
MoE-Study — Dense vs. Mixture-of-Experts, matched active parameters
Two decoder-only language models trained from scratch under identical conditions, differing in exactly one thing: whether the feed-forward block is a dense MLP or a sparse top-2-of-4 MoE.
Both checkpoints live in this one repo:
The MoE's active-parameter count matches Dense by construction — 2 of 4 experts at half the hidden size means identical compute per token. The MoE only spends more memory for extra capacity.
Full write-up, training code, and evaluation harness: [github.com/OliverSundaram/MoE-Study](https://github.com/OliverSundaram/MoE-Study)
⚠️ These are research artifacts, not usable models
Read this before downloading.
- Trained for one epoch on ~40.7M tokens — neither model is close to converged.
- WikiText word perplexity is 551 (Dense) and 1,378 (MoE). Generations are largely incoherent.
- 0.0% on LAMBADA for both — at the task floor.
- No instruction tuning, no RLHF, no safety filtering of any kind.
They exist to answer one narrow question: at matched active compute and matched budget, does sparsity help? They are not fit for any downstream use.
Getting the weights
These are a custom architecture, not a variant of an existing one. The modeling code is not included here, so from_pretrained on this repo alone will not build the model.
Clone the GitHub repo — it carries the model definition and loading instructions, and points back at these subfolders for the weights.
Model details
Shared architecture
Both models are the same custom decoder-only transformer:
The one difference
Both models share the same unmodified GPT-2 tokenizer, stored once at the repo root.
Training
Identical for both models. Single consumer GPU, no cloud.
Final losses
Dense has the lower loss at every checkpoint.
Evaluation
All benchmarks via lm-evaluation-harness on the final checkpoints.
How to read this:
- Dense wins on everything sensitive to raw LLM quality — perplexity, ARC-Easy, PIQA.
- WinoGrande, HellaSwag, and ARC-Challenge are won by MoE, but with such a negligible difference, that they could be considered to have an equal accuracy
Inference speed
Greedy decoding, 32-token prompt → 64 new tokens, 5 trials, 2 warmup, no KV cache.
MoE is ~3.1× slower despite matched active compute — an artifact of unoptimized expert dispatch, not a property of the architecture.
<details> <summary><b>Benchmark charts</b></summary>
</details>
Findings
1. Dense won every metric that wasn't already at chance. Most clearly on WikiText perplexity — 551 vs 1,378, a 2.5× gap.
2. The routing math is correct. Active parameters match Dense almost exactly. Matched active compute simply didn't buy matched quality at this budget.
3. Routing stayed balanced. The load-balancing term sat on its theoretical floor, so the MoE's gap is not explained by experts collapsing onto each other.
4. Extra capacity needs extra tokens. The MoE has 38% more parameters but saw the same ~40.7M tokens — likely far too few to train 4 experts per layer, each seeing only a routed fraction of the stream. ---
Citation
@misc{sundaram2026moestudy,
author = {Sundaram, Oliver},
title = {MoE-Study: Dense vs. Mixture-of-Experts at Matched Active Parameters},
year = {2026},
url = {https://github.com/OliverSundaram/MoE-Study}
}Acknowledgments
- lm-evaluation-harness (EleutherAI) — evaluation
- nampdn-ai/tiny-textbooks — training corpus
- Hugging Face `transformers` — base classes and tokenizer
License
MIT