CoolFace
Modelpublic

ShahzebKhoso/LocateAnything-3B-GGUF

sourceHugging Faceotherupdated 4mo agoView on Hugging Face
1likes250downloads
Model Card

LocateAnything-3B - GGUF Quantized Versions

This repository provides GGUF quantized versions of nvidia/LocateAnything-3B, converted with llama.cpp.

The purpose of this repository is to provide fast, easy-to-use local inference files for llama.cpp, Ollama, LM Studio, Jan, Open WebUI, and llama-cpp-python users.

Model Details

  • —Base model: nvidia/LocateAnything-3B
  • —Architecture: Transformer
  • —Format: GGUF
  • —Source license: other
  • —Conversion tool: convert_hf_to_gguf.py from llama.cpp
  • —Quantization tool: llama-quantize
  • —Recommended file: LocateAnything-3B-Q4_K_M.gguf

Quantized Files

QuantFilenameSizeSHA256Notes
FP16LocateAnything-3B-FP16.gguf~6.34 GiBa4c0b3ebe23f...Full precision converted GGUF baseline
Q2_KLocateAnything-3B-Q2_K.gguf~1.28 GiB6569e7b98cd3...Smallest, lowest quality
Q3KMLocateAnything-3B-Q3_K_M.gguf~1.61 GiBbdfbe9fe0b80...Small balanced version
Q4_0LocateAnything-3B-Q4_0.gguf~1.86 GiBd42738b1cb53...Simple 4-bit quantization
Q4KMLocateAnything-3B-Q4_K_M.gguf~1.96 GiBd06f4263b54f...Recommended default for most users
Q5KMLocateAnything-3B-Q5_K_M.gguf~2.27 GiB7177002d8228...Better quality with moderate size
Q6_KLocateAnything-3B-Q6_K.gguf~2.60 GiBbd0775eb3c08...High quality
Q8_0LocateAnything-3B-Q8_0.gguf~3.37 GiB11d4ab968014...Near FP16 quality

Validation

Each file was tested with llama-cli for basic load + generation.

QuantFilenameStatus
FP16LocateAnything-3B-FP16.gguf✅ passed
Q2_KLocateAnything-3B-Q2_K.gguf✅ passed
Q3KMLocateAnything-3B-Q3_K_M.gguf✅ passed
Q4_0LocateAnything-3B-Q4_0.gguf✅ passed
Q4KMLocateAnything-3B-Q4_K_M.gguf✅ passed
Q5KMLocateAnything-3B-Q5_K_M.gguf✅ passed
Q6_KLocateAnything-3B-Q6_K.gguf✅ passed
Q8_0LocateAnything-3B-Q8_0.gguf✅ passed

Usage

llama.cpp

bash
llama-cli -m LocateAnything-3B-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

Older builds may use:

bash
./main -m LocateAnything-3B-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

llama.cpp directly from Hugging Face

bash
llama-cli -hf ShahzebKhoso/LocateAnything-3B-GGUF:Q4_K_M -p "Hello! Introduce yourself briefly."

llama-cpp-python

python
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="ShahzebKhoso/LocateAnything-3B-GGUF",
    filename="LocateAnything-3B-Q4_K_M.gguf",
)

llm = Llama(model_path=model_path)

out = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello! Introduce yourself briefly."},
    ],
    max_tokens=128,
)

print(out["choices"][0]["message"]["content"])

Which file should I use?

  • —Use Q4_K_M for the best default balance.
  • —Use Q5_K_M for better quality.
  • —Use Q8_0 if you want near-original quality and have more memory.
  • —Use Q2_K or Q3_K_M only when memory is very limited.

Provenance

This repository is a quantized derivative of:

text
nvidia/LocateAnything-3B

Base model metadata:

text
revision: 7a81d810571dc5f244b2f0b6868128f24b1cbd85
pipeline_tag: image-text-to-text
tags: transformers, safetensors, locateanything, feature-extraction, nvidia, eagle, vision, object-detection, grounding, arxiv:2605.27365, image-text-to-text, conversational, custom_code, en, arxiv:2504.07491, arxiv:2109.10852, arxiv:2510.12798, arxiv:2303.05499, arxiv:1405.0312, arxiv:1908.03195