Modelpublic

lavawolfiee/Mixtral-8x7B-Instruct-v0.1-offloading-demo

sourceHugging Facemitupdated 3y agoView on Hugging Face

27likes89downloads

Model Card

Attention quantization: HQQ 4-bit, groupsize 64, compress zero, compress scale with groupsize 256 \ Experts quantization: HQQ 2-bit, groupsize 16, compress zero, compress scale with groupsize 128