hoangton/gemma-2b-GGUF

Gemma-2B GGUF

This is a quantized version of the google/gemma-2b model using llama.cpp.

This model card corresponds to the 2B base version of the Gemma model. You can also visit the model card of the 7B base model, 7B instruct model, and 2B instruct model.

Model Page: Gemma

Terms of Use: Terms

⚡ Quants

• —q2_k: Uses Q4K for the attention.vw and feedforward.w2 tensors, Q2_K for the other tensors.
• —q3_k_l: Uses Q5K for the attention.wv, attention.wo, and feedforward.w2 tensors, else Q3_K
• —q3_k_m: Uses Q4K for the attention.wv, attention.wo, and feedforward.w2 tensors, else Q3_K
• —q3_k_s: Uses Q3_K for all tensors
• —q4_0: Original quant method, 4-bit.
• —q4_1: Higher accuracy than q40 but not as high as q50. However has quicker inference than q5 models.
• —q4_k_m: Uses Q6K for half of the attention.wv and feedforward.w2 tensors, else Q4_K
• —q4_k_s: Uses Q4_K for all tensors
• —q5_0: Higher accuracy, higher resource usage and slower inference.
• —q5_1: Even higher accuracy, resource usage and slower inference.
• —q5_k_m: Uses Q6K for half of the attention.wv and feedforward.w2 tensors, else Q5_K
• —q5_k_s: Uses Q5_K for all tensors
• —q6_k: Uses Q8_K for all tensors
• —q8_0: Almost indistinguishable from float16. High resource use and slow. Not recommended for most users.

💻 Usage

This model can be used with the latest version of llama.cpp and LM Studio >0.2.16.