Modelpublic

lavawolfiee/Mixtral-8x7B-Instruct-v0.1-offloading-hqq-4bit-3bit

sourceHugging Facemitupdated 3y agoView on Hugging Face

1likes16downloads

Model Card

Attention quantization: HQQ 4-bit, groupsize 64, compress zero, compress scale with groupsize 256 \ Experts quantization: HQQ 3-bit, groupsize 64, compress zero, compress scale with groupsize 128