jusjinuk/Llama-2-7b-hf-3bit-GuidedQuant-QTIP

Model Card

• —Base model: meta-llama/Llama-2-7b-hf
• —Quantization method: BlockLDLQ with GuidedQuant Hessian
• —Target bit-width: 3
• —Backend kernel: QTIP kernel (HYB variant)
• —Calibration data: RedPajama (1024 sentences / 4096 tokens)
• —Calibration objective: Next-token prediction
• —num_groups (for GuidedQuant Hessian): 4

How to run

• —Follow the instruction in https://github.com/snu-mllab/GuidedQuant and https://github.com/Cornell-RelaxML/qtip

References

• —Model Paper