CoolFace
Modelpublic

hotdogs/frankenmoe

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
1likes53downloads
Model Card

🧪 FrankenMoE — Proof of Concept (NOT production)

This is a technical experiment, not a useful model.

⚠️ Important Warning

This repository documents a proof-of-concept MoE pipeline. The model quality is NOT good — it produces incoherent / random outputs because:

  1. 1.The router uses (no training)
  2. 2.Experts were fine-tuned with only ~5K samples each
  3. 3.Base model is only Qwen2.5-1.5B-Instruct

Do NOT use this model for anything serious. It exists purely to demonstrate that the FrankenMoE pipeline can be built end-to-end.

What We Actually Built

A working MoE pipeline from dense LoRA experts → GGUF:

Qwen2.5-1.5B-Instruct (base)
  ├── Expert 0: Coding (LoRA fine-tuned)
  ├── Expert 1: Math (LoRA fine-tuned)  
  └── Shared Expert: Base model

Key Technical Discoveries

DiscoveryDetail
mergekit 0.1.4 bugparam incompatible with transformers >= 4.40 — must patch
QwenMoE requirementsExactly 1 shared expert + 2^n routed experts (2, 4, 8)
Tied embeddings fixQwen2.5 uses tied embeddings → must clone → before GGUF conversion, set
LoRA must be mergedAdapters must be before MoE assembly

Repository Structure

📦 frankenmoe_moe_v2-F16.gguf  — MoE GGUF (fixed, has output.weight)
📁 moe_full/                   — Full safetensors model
📁 coding/ math/ chat/         — Individual dense experts (LoRA + GGUF)
📄 FrankenMoE_Academic_Paper.pdf — Research paper
🐍 simple_router.py            — Keyword-based router (functional alternative)

Quick Test

bash
wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/frankenmoe_moe_v2-F16.gguf
llama-cli -m frankenmoe_moe_v2-F16.gguf -p "Write a Python function"
# Output: Random/incoherent — this is expected! See warning above.

Future: Real Model

The pipeline will be re-run with:

  • —Larger base model (Qwen2.5-7B/14B)
  • —Trained router (classification loss)
  • —More training data per domain
  • —4 experts for proper 2^n routing

Stay tuned — the real model is coming.

Built by UKA 🇹🇭 | May 2026