CoolFace
Modelpublic

majentik/harrier-oss-v1-0.6b-GGUF-Q5_K_M

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes16downloads
Model Card
[!TIP] KV-cache quantization (upstream, no fork needed): llama.cpp/Ollama cover this natively — -ctk q8_0 -ctv q8_0 (~half KV memory, negligible quality loss) or -ctk q4_0 -ctv q4_0 (~quarter memory, small quality cost). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1.

harrier-oss-v1-0.6b GGUF Q5KM

llama.cpp GGUF Q5KM quantization of microsoft/harrier-oss-v1-0.6b.

  • —Produced with: llama-quantize (upstream llama.cpp)
  • —BF16 source via convert_hf_to_gguf.py
  • —Quant type: Q5_K_M
  • —File size: 424 MB

Quickstart

bash
llama-embedding -m harrier-0.6b-Q5_K_M.gguf -p "What is Harrier-OSS?"

License

MIT — inherited from the upstream Harrier-OSS-v1-0.6B.

See also