Modelpublic

atfai/granite-embedding-97m-multilingual-r2-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes224downloads

Model Card

granite-embedding-97m-multilingual-r2-GGUF

F16 GGUF conversion of `ibm-granite/granite-embedding-97m-multilingual-r2` for local serving with llama.cpp. Converted and independently verified by ATF (Agent Taskflow) for edge-local embedding serving via atf-serve.

This is a format conversion only — no weights were modified, retrained, or fine-tuned. All model weights are © IBM, licensed Apache-2.0 (same as the base model). This repository is not affiliated with or endorsed by IBM.

Why this exists

IBM does not publish a GGUF for this model. This repo documents its own build end-to-end — source checksum, conversion command, and independent correctness verification — rather than asking you to trust an unverified re-hosted binary.

Conversion details

—Source: ibm-granite/granite-embedding-97m-multilingual-r2, model.safetensors (bf16)
—Tool: llama.cpp built from source at commit 11924d4c17abc27383376a1ac6a24fa3e36c1c0c (2026-08-02). This model uses its own purpose-trained 180,000-token multilingual tokenizer — a distinct tokenizer from the 311m sibling model, registered upstream as granite-embed-multi-97m (a different pre-tokenizer type than the 311m variant's granite-embed-multi-311m/GEMMA4). Neither registration is recognized by llama.cpp release b9204 or earlier; a current build is required both to convert and to serve this model.
—Command:

  python3 convert_hf_to_gguf.py <model-dir> \
    --outfile granite-embedding-97m-multilingual-r2-f16.gguf \
    --outtype f16

—Output: F16, 384-dim, 206,403,072 bytes.

Verification (independent, not vendor-claimed)

Embedded the same test sentence through both this GGUF (via llama-server --embedding --pooling cls) and the original HF model (via sentence-transformers, loaded directly from the source safetensors), then computed cosine similarity between the two output vectors. Verified on both CPU and GPU (CUDA) inference paths.

Check	Result
Output dimension	384 (matches source `hidden_size`)
Cosine similarity vs. HF reference pipeline (CPU)	0.999980
Cosine similarity vs. HF reference pipeline (GPU/CUDA)	0.999980
Required pooling mode	`cls` (matches source `classifier_pooling: "cls"` in `config.json`; mean pooling is not correct for this model)
CPU footprint (process RSS, mmap'd)	~299 MB
GPU VRAM (full offload, `-ngl 99`)	~354 MB
Latency (single request, CPU / GPU)	~23 ms / ~10 ms

See the sibling `granite-embedding-311m-multilingual-r2-GGUF` for the larger variant.

Usage

llama-server --model granite-embedding-97m-multilingual-r2-f16.gguf \
  --embedding --pooling cls --port 8089

Requires a llama.cpp build that includes granite-embed-multi-97m tokenizer support (see Conversion details above — current upstream master has it; check your pinned release tag if serving fails with an unknown pre-tokenizer type error).

curl http://127.0.0.1:8089/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "your text here", "model": "granite-embedding-97m"}'

Converted by ATF — agent orchestration with edge-local model serving.