Modelpublic

sayhan/gemma-7b-GGUF-quantized

sourceHugging Faceotherupdated 3y agoView on Hugging Face

1likes290downloads

Model Card

image/webp

Google Gemma 7B

—Model creator: Google
—Original model: gemma-7b-it
—**Terms of use**

Description

This repo contains GGUF format model files for Google's Gemma 7B

Original model

—Developed by: Google

Description

Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. They are text-to-text, decoder-only large language models, available in English, with open weights, pre-trained variants, and instruction-tuned variants. Gemma models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop or your own cloud infrastructure, democratizing access to state of the art AI models and helping foster innovation for everyone.

Quantizon types

quantization method	bits	size	description	recommended
Q2_K	2	3.09	very small, very high quality loss	❌
Q3KS	3	3.68 GB	very small, high quality loss	❌
Q3KL	3	4.4 GB	small, substantial quality loss	❌
Q4_0	4	4.81 GB	legacy; small, very high quality loss	❌
Q4KS	4	4.84 GB	medium, balanced quality	✅
Q4KM	4	5.13 GB	medium, balanced quality	✅
Q5_0	5	5.88 GB	legacy; medium, balanced quality	❌
Q5KS	5	5.88 GB	large, low quality loss	✅
Q5KM	5	6.04 GB	large, very low quality loss	✅
Q6_K	6	7.01 GB	very large, extremely low quality loss	❌
Q8_0	8	9.08 GB	very large, extremely low quality loss	❌
FP16	16	17.1 GB	enormous, negligible quality loss	❌

Usage

You can use this model with the latest builds of LM Studio and llama.cpp. If you're new to the world of large language models, I recommend starting with LM Studio.