Modelpublic

atfai/granite-embedding-311m-multilingual-r2-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes52downloads

Model Card

granite-embedding-311m-multilingual-r2-GGUF

F16 GGUF conversion of `ibm-granite/granite-embedding-311m-multilingual-r2` for local serving with llama.cpp. Converted and independently verified by ATF (Agent Taskflow) for edge-local embedding serving via atf-serve.

This is a format conversion only — no weights were modified, retrained, or fine-tuned. All model weights are © IBM, licensed Apache-2.0 (same as the base model). This repository is not affiliated with or endorsed by IBM.

Why this exists

IBM does not publish a GGUF for this model. This repo documents its own build end-to-end — source checksum, conversion command, and independent correctness verification — rather than asking you to trust an unverified re-hosted binary.

Conversion details

—Source: ibm-granite/granite-embedding-311m-multilingual-r2, model.safetensors (bf16, 623,341,952 bytes)
—Tool: llama.cpp built from source at commit 11924d4c17abc27383376a1ac6a24fa3e36c1c0c (2026-08-02). This model's tokenizer (granite-embed-multi-311m, maps to LLAMA_VOCAB_PRE_TYPE_GEMMA4) is not recognized by llama.cpp release b9204 or earlier — the registration landed upstream after that tag. A current build (or any release ≥ the commit that added it) is required both to convert and to serve this model; older binaries fail with unknown pre-tokenizer type: 'granite-embed-multi-311m' at load time, not at conversion time.
—Command:

  python3 convert_hf_to_gguf.py <model-dir> \
    --outfile granite-embedding-311m-multilingual-r2-f16.gguf \
    --outtype f16

—Output: F16, 768-dim, 638,121,344 bytes.

Verification (independent, not vendor-claimed)

Embedded the same test sentence through both this GGUF (via llama-server --embedding --pooling cls) and the original HF model (via sentence-transformers, loaded directly from the source safetensors), then computed cosine similarity between the two output vectors.

Check	Result
Output dimension	768 (matches source `hidden_size`)
Cosine similarity vs. HF reference pipeline	0.999970
Required pooling mode	`cls` (matches source `classifier_pooling: "cls"` / `pooling_mode_cls_token: true` in `config.json`; mean pooling is not correct for this model)

Usage

llama-server --model granite-embedding-311m-multilingual-r2-f16.gguf \
  --embedding --pooling cls --port 8089

Requires a llama.cpp build that includes granite-embed-multi-311m tokenizer support (see Conversion details above — current upstream master has it; check your pinned release tag if serving fails with an unknown pre-tokenizer type error).

curl http://127.0.0.1:8089/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "your text here", "model": "granite-embedding-311m"}'

Converted by ATF — agent orchestration with edge-local model serving.