Modelpublic

NexesQuants/mistral-small-3.1-24b-instruct-2503-iMat-IKLQ-GGUF

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes81downloads

Model Card

About

IQ2KS, IQ4KS, IQ5KS are Gen 2 IQK Quants from Ikawrakow. They are faster (PP for sure, TG maybe) than the gen 1 IKQuants (IQ2K to IQ6_K). They'll work with my last release of Croco.Cpp.

https://github.com/Nexesenex/croco.cpp/releases/tag/v1.92055b5145RM1.102

And of course on IK_Llama.cpp

Cuda Pascal or more recent GPU needed, I didn't adapt or compile for anything else.