Koshkasa/Naphula_Slimaki-Tavern-24B-v1.3_IQ4_KT-GGUF

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This is a IQ4KT quantization of [Naphula/Slimaki-Tavern-24B-v1.3](https://huggingface.co/Naphula/Slimaki-Tavern-24B-v1.3), created specifically to challenge mainline IQ4XS quants by being smaller and not being dumber.

Quant Details

iq4kt: attnq, attnv, attnk, ffndown, ffngate, ffnup, attnoutput (280) iq4k: tokenembd (1) iq6_k: output (1) imatrix by mradermacher

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp only tested with ik_llama.cpp (though I expect croco.cpp to work as well?)

Comparison:

IQ4_XS by [mradermacher](https://huggingface.co/mradermacher/Slimaki-Tavern-24B-v1.3-i1-GGUF):

winogrande: 74.3489 ± 1.2274 (942/1267) hellaswag (n=400, s=42): 82.25%

with K q50, V q41: pp512: 1457.63 ± 157.65 pp32768: 1108.26 ± 5.68 tg128: 31.27 ± 0.79 tg512: 31.54 ± 0.16 size: 12758946304 bytes

this IQ4_KT:

winogrande: 74.7435 ± 1.2211 (947/1267) (+0.53%, statistically insignificant) hellaswag (n=400, s=42): 83.25% (+1.2%, statistically insignificant)

with K q50, V q41: pp512: 1168.07 ± 30.28 (19.8% slower) pp32768: 955.13 ± 4.38 (13.8% slower) tg128: 32.39 ± 0.22 (3.58% faster) tg512: 31.52 ± 0.26 (0.07% slower) size: 12070818560 bytes (5.39% smaller) (-656 MB)

In conclusion

-Is it smarter? Not really.

-Is it slower? Yeah.

-Is it smaller? Yeah. 656 MB is about 12k extra context at K(Q5_0) V(Q4_1).

-Is it worth it? No idea. I made it for personal use to squeeze some extra context in without losing brainpower. I can take the PP speed hit. WYSIWYG.

-Possible 12G VRAM full fit with KVO? Probably. A mixed IQ3_KT ffn / IQ4_KT attn would likely fit with some context headroom. It's going to be slower. It's probably not going to be THAT much dumber.

-Why Q5_0/Q4_1? [Reasons](https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context)

Cheers

[MistralAI](https://huggingface.co/mistralai) - the beloved base model(s). [ikawrakow and contributors of ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) - I probably misused your wonderful creation. [Naphula](https://huggingface.co/Naphula) - you cooked a sick one. Everyone whose finetunes were included in the merge! [mradermacher](https://huggingface.co/mradermacher) - for the imatrix + the myriad of quants we all benefit from.