embedme/lightonai-colbert-zero-f16

ColBERT-Zero (GGUF f16 + Projection)

Quantized GGUF conversion of lightonai/ColBERT-Zero for use with litembeddings.

ColBERT-Zero is SOTA on BEIR (55.43 nDCG@10) for models under 150M parameters, outperforming all other ColBERT and dense retrieval models trained on public data.

Model Details

Available Variants

Benchmarked on QEMU vCPU with SSE4.2. Q8_0 is fastest due to integer SIMD; f16 is slowest without hardware FP16.

BEIR Benchmark (from original model)

MaxSim Score Consistency Across Quants

Negligible quality loss from quantization — Q8_0 scores within 0.1% of f32.

Files

Usage with litembeddings

.load ./build/litembeddings

-- Load model with projection
SELECT lembed_model('lightonai-colbert-zero-f16.gguf',
    '{"colbert_projection": "lightonai-colbert-zero-f16.projection"}');

-- Generate token embeddings
SELECT lembed_tokens('search_query: What is machine learning?');

-- Semantic search with MaxSim scoring
SELECT
    id, content,
    lembed_maxsim(lembed_tokens('search_query: error handling best practices'), tokens) AS score
FROM documents
ORDER BY score DESC
LIMIT 10;

Important: Query/Document Prefixes

ColBERT-Zero uses asymmetric prompts for best results:

• —Queries: Prefix with search_query:
• —Documents: Prefix with search_document:

Omitting these prefixes degrades performance by ~0.8-1.3 nDCG@10 points.

Conversion

python scripts/convert_colbert_to_gguf.py lightonai/ColBERT-Zero ./models \
    --name colbert-zero --quantize f16

License: Apache 2.0