Modelpublic

Koshkasa/sophosympatheia_Magistry-24B-v1.1-IQ4_KS.gguf

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes33downloads

Model Card

What's that?

The goal: Make a medium quality quant of sophosympatheia/Magistry-24B-v1.1 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead. The result: Mixed precision quantization of sophosympatheia/Magistry-24B-v1.1

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp

Layout

Layer	Dims	Dims	Quant
token\_embd	5120	131072.0	iq4\_k

	blk	40
attn\_k	5120	1024	iq6\_k
attn\_norm	5120	1	f32
attn\_q	5120	4096	iq6\_k
attn\_v	5120	1024	iq6\_k
ffn\_down	32768	5120	iq4\_k
ffn\_gate	5120	32768	iq4\_ks
ffn\_norm	5120	1	f32
ffn\_up	5120	32768	iq4\_ks
attn\_output	4096	5120	iq6\_k

output	5120	131072	iq6\_k
output\_norm	5120	1	f32

Cheers

[MistralAI](https://huggingface.co/mistralai) - the beloved base model(s). [ikawrakow and contributors of ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) - I probably misused your wonderful creation. [sophosympatheia](https://huggingface.co/sophosympatheia) - for the merge effort. Everyone whose finetunes were included in the merge! [bartowski](https://huggingface.co/bartowski) - for the imatrix + the myriad of quants we all benefit from.