thlurte/all-MiniLM-L6-v2-GGUF

all-MiniLM-L6-v2-GGUF

This repository contains GGUF format model files for sentence-transformers/all-MiniLM-L6-v2, generated with the intextus BERT GGUF conversion pipeline.

These files are fully compatible with llama.cpp and intextus.

Available Files & Quantizations

• —all-MiniLM-L6-v2-F32.gguf: Full precision F32 (recommended for maximum accuracy and zero dequantization overhead on CPU).
• —all-MiniLM-L6-v2-F16.gguf: Half precision F16.
• —all-MiniLM-L6-v2-Q8_0.gguf: 8-bit quantization (recommended default, fast and near-lossless).
• —all-MiniLM-L6-v2-Q6_K.gguf: 6-bit quantization.
• —all-MiniLM-L6-v2-Q5_K_M.gguf: 5-bit quantization (Medium).
• —all-MiniLM-L6-v2-Q5_0.gguf: 5-bit quantization.
• —all-MiniLM-L6-v2-Q4_K_M.gguf: 4-bit quantization (Medium).
• —all-MiniLM-L6-v2-Q4_0.gguf: 4-bit quantization.
• —all-MiniLM-L6-v2-Q3_K_M.gguf: 3-bit quantization (Medium).
• —all-MiniLM-L6-v2-Q2_K.gguf: 2-bit quantization.