Modelpublic

Muzian/minimind-rerank-moe

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes16downloads

Model Card

MiniMind-Rerank-MoE

</div>

A Mixture-of-Experts cross-encoder reranker built on the MiniMind-3-MoE backbone (198M total / 64M active). The MoE counterpart of MiniMind-Rerank-Dense.

📌 How It Works

Same pointwise yes/no design as the Dense version: given a query and document, the model predicts whether the next token is 是 (relevant) or 否 (irrelevant). The softmax probability of 是 is the relevance score.

📊 Evaluation

C-MTEB/T2Reranking (MAP@10, no data leakage)

[image]

</div>

Model	MAP@10	Accuracy
Rerank-Dense (64M)	0.915	85%
Rerank-MoE (198M-A64M)	0.604	88%

Key Finding: MoE classification > Dense, but ranking < Dense

💡 Counterintuitive: MoE's training accuracy (88%) is higher than Dense (85%), yet its MAP@10 (0.604) is far lower (Dense: 0.915). Why? - Pointwise ≠ Ranking: MoE excels at binary classification (yes/no), but ranking requires fine-grained discrimination between "very relevant" vs "somewhat relevant." MoE's top-1 routing fragments the sequence across experts, making it harder to form a coherent global relevance signal. - Consistent with embedding findings: just as MoE underperforms Dense on STS (0.422 vs 0.478), the MoE architecture hurts representation quality at small scales. Recommendation: for reranking, use MiniMind-Rerank-Dense — smaller, faster, and scores 50% higher.

🎓 Training

Config	Value
Training data	T2Reranking, 80k pairs (80/20 split, no leakage)
Epochs	3
Batch size	16
Learning rate	1e-5 (cosine)
Final loss	0.28
Final accuracy	88%

🏗️ Model Details

Item	Value
Architecture	MiniMind-3-MoE (4 experts, top-1)
Total params	198M
Active params	64M
Scoring	pointwise yes/no
Target tokens	`是` (357) / `否` (1332)

🙏 Acknowledgements

—Base: MiniMind by @jingyaogong
—Route: Qwen3-Reranker
—Data: C-MTEB/T2Reranking

📄 License

Apache 2.0