CoolFace
Modelpublic

Kquant03/Hippolyta-7B-GGUF

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face
0likes97downloads
Model Card

image/jpeg

The flower of Ares.

These are the GGUF files of the fine-tuned model. To be compiled with llama.cpp on oobabooga or VLLm.

Fine-tuned on mistralai/Mistral-7B-v0.1...my team and I reformatted many different datasets and included a small amount of private stuff to see how much we could improve mistral.

I spoke to it personally for about an hour, and I believe we need to work on our format for the private dataset a bit more, but other than that, it turned out great. I will be uploading it to open llm evaluations, today.

Provided files

NameQuant methodBitsSizeMax RAM requiredUse case
Q2_K TinyQ2_K22.7 GB4.7 GBsmallest, significant quality loss - not recommended for most purposes
Q3_K_MQ3KM33.52 GB5.52 GBvery small, high quality loss
Q4_0Q4_044.11 GB6.11 GBlegacy; small, very high quality loss - prefer using Q3KM
Q4_K_MQ4KM44.37 GB6.37 GBmedium, balanced quality - recommended
Q5_0Q5_055 GB7 GBlegacy; large, balanced quality
Q5_K_MQ5KM55.13 GB7.13 GBlarge, balanced quality - recommended
Q6 XLQ6_K65.94 GB7.94 GBvery large, extremely low quality loss
Q8 XXLQ8_087.7 GB9.7 GBvery large, extremely low quality loss - not recommended
  • —Uses Mistral prompt template with chat-instruct.