Modelpublic

Koshkasa/Vortex5_Shadow-Siren-26B-A4B-APEX-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

6likes436downloads

Model Card

What's that?

As denoted by its creators, APEX is "a novel MoE-aware mixed-precision quantization technique for llama.cpp". I suggest reading their information on the how and why of the method.

These two GGUFs are APEX quantizations of Vortex5/Shadow-Siren-26B-A4B, using EmanuelOverride's custom APEX recipe, HarmonicMinus, and a minor customization of that edit to cap shared ffn at Q6K and replace Q4K and Q40 (which was a fallback for ffndownexps from Q3K) sparse expert quants with MXFP4. Both recipes use mradermacher's imatrix found here

To quote EmanuelOverride regarding the philosophy behind HarmonicMinus:

🎵 The Harmonic Series (Pulsed Attention)

Following on from the Harmonic Resonance branch, the Harmonic layouts further optimize layer contiguity for performance, by following strict precision tiers across all 30 layers (High/Medium/Low), while priorizing high-precision tiers for Global Attention (GA) layers, preventing aggressive compression in order to anchor deep-context tracking.

—[i-Harmonic.Minus](https://huggingface.co/EmanuelOverride/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/gemma-4-26B-A4B-it-heretic.APEX.i.Harmonic.Minus.gguf): 14,369.74 MiB (4.78 BPW) — High-performance configuration. Condenses the architecture into 3 strict tiers (5/15/10). Shifts high-precision blocks to coincide with global attention pulses. -------------------------------- My modification, aptly named HarmonicMinusMinus, retains most of the original recipe and adheres to the same philosophy, sacrificing some precision on sparse experts FFN layers for extra VRAM savings, while retaining global attention-focused attn distribution. A secondary effect is minor speedup on GPUs with native fp4 support. The HarmonicMinusMinus modification APEX recipe is provided.

Benchmarks

Measurement	HarmonicMinus	HarmonicMinusMinus	Delta
pp512	2764.56	3233.05	+16.97%
tg128	106.81	112.63	+5.45%
pp32768	2378.52	2543.86	+6.95%
tg512	102.54	103.86	+1.29%
pp32768 + tg512 wall time (s)	18.76	17.81	-5.06%
hellaswag% (s=42, n=400, greedy)	52.25%	52.25%	==
hellaswag (s=42, n=400, greedy)	209	209	==
winogrande% (n=1267, greedy)	55.09%	57.06%	+1.97% (abs)
winogrande (n=1267, greedy)	698	723	+25
Size (bytes)	15083595360	14157258080	-6.14%
Bits per weight	4.78	4.48	-0.3

pp32768/tg512 tests were run at q40/q40 cache quantization to ensure both models fully fit in VRAM with KV cache on my GPU.

Conclusion

WYSIWYG. MXFP4 is lower precision than Q4K and can underrepresent outliers. For certain ffndown_exps it may have had an impact that had not been identified during my surface testing. Feedback welcome.

Disclosure

My only contribution is compute and a little tinkering with an already existing custom APEX recipe. This is neither my merge, nor my quantization methodology. Have fun.

Cheers

[Google](https://huggingface.co/google) - the base model. [Vortex5](https://huggingface.co/Vortex5) - for the merge effort. Everyone whose finetunes were included in the merge! [mudler](https://github.com/localai-org/apex-quant) - for the APEX quantization methodology. [EmanuelOverride](https://huggingface.co/EmanuelOverride) - for the custom APEX experiments and the base recipe. [mradermacher](https://huggingface.co/mradermacher) - for the imatrix + myriads of quants we all benefit from.