Modelpublic

noctrex/Ling-3.0-tiny-MXFP4_MOE-GGUF

sourceHugging Faceupdated 1mo agoView on Hugging Face

2likes12kdownloads

Model Card

These are MXFP4 quantizations of the model inclusionAI / Ling-3.0-tiny

Quick Start

1.Download the latest release of **llama.cpp**.
2.Download your preferred model variant from below.

Which version should I choose?

All FP4 variants use MXFP4 for the MoE (Mixture of Experts) weights to keep the model efficient. I've included also a new type Q8XLMOE, that uses Q8 for MoE tensors and BF16 for everything else. The difference lies in how the remaining tensors are handled:

Variant	Quality	Performance	Size	Recommendation
Q8_XL_MOE	⭐⭐⭐⭐⭐	Variable*	8.77GiB	Maximum quality, uses Q8 instead of FP4 for the MoE weights.
BF16	⭐⭐⭐	Variable*	4.54GiB	Best for maximum accuracy; original unquantized weights.
F16	⭐⭐	Fast	4.94GiB	Great alternative if BF16 is slow on your hardware.
Q8	⭐	Fastest	4.94GiB	Balanced performance and memory usage.

Note: On some older architectures, BF16 may be slower than F16. Check that your GPU supports native BF16

Recommended parameters from inclusionAI:

—temperature=1.0
—top_p=0.95
—top_k=20