Modelpublic

PatronusAI/glider-gguf

sourceHugging Facecc-by-nc-4.0updated 2y agoView on Hugging Face

3likes43downloads

Model Card

Available GGUF versions for the PatronusAI/glider model: [BF16, Q8_0, Q5_K_M, Q4_K_M]

How to load your desired quantized model:

1.Select the appropraite GGUF quantization from the available list above
2.Run the following code:

bash

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("PatronusAI/glider-gguf", gguf_file="glider_{version_from_list}.gguf")

For loading the Q8_0 version, this script will change to:

bash

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("PatronusAI/glider-gguf", gguf_file="glider_Q8_0.gguf")

For any issues or questions, reach out to Darshan Deshpande or Rebecca Qian