mudler/locate-anything.cpp-gguf
locate-anything.cpp - GGUF
GGUF builds of `nvidia/LocateAnything-3B` for [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) - a C++/ggml inference engine for open-vocabulary detection / visual grounding, no Python at inference time.
Brought to you by the [LocalAI](https://github.com/mudler/LocalAI) team.
The detections are the same as the official PyTorch implementation (the engine is parity-gated against it), and it runs faster - on CPU and GPU.
Files
The full-precision f32 GGUF (~15 GB) is reproducible from the HF weights with scripts/convert_locateanything_to_gguf.py in the repo.
Performance
Same detections as the official model, faster. Full methodology, the warm/median setup, parity checks, and more images are in the repo's `benchmarks/BENCHMARK.md`.
Quantization (CPU, Ryzen 9 9950X3D)
Slow-mode inference on the 448 fixture; vs official divides the official PyTorch f32 time (23.65 s) by each. Only the Qwen2 LM matmuls are quantized, so box parity is preserved through q6_k:
GPU (NVIDIA GB10, vs the official bf16 model)
Run against the official model exactly as its model card documents (bf16), greedily, on one GB10 GPU. Precision-matched (our f16 vs its bf16) ours is ~1.7× faster; the recommended q8_0 build (box-identical) is ~1.9-2.1×:
Quantization policy
Only the Qwen2 language-model matmuls (attn_{q,k,v,o}, ffn_{gate,up,down}, lm.output) are quantized. The MoonViT vision tower, the projector, all norms and biases, and the two host-read f32 tensors (lm.tok_embd, vit.pos_emb) stay f32 - so the parity-sensitive vision path is untouched. q80/q6k are box-identical; lower bit-widths trade a little box precision for size.
Usage
# build the CLI (see the repo README), then:
locate-anything-cli detect \
--model locate-anything-q8_0.gguf \
--input image.jpg \
--prompt "Locate all the instances that matches the following description: person</c>car." \
--annotated out.png
# -> {"detections":[{"label":"person","box":[...]}, ...]} + an annotated PNGDecode modes: --mode hybrid (default), slow, fast. GPU: build with -DLA_GGML_CUDA=ON and run with LA_DEVICE= (auto-GPU). Separate categories in the prompt with </c>.
License
The model weights are NVIDIA's, distributed under NVIDIA's license; this repository redistributes them in GGUF form for use with locate-anything.cpp (MIT).