Modelpublic

Gramscii-IT/mmarco-mMiniLMv2-L12-H384-v1-GGUF

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes372downloads

Model Card

mmarco-mMiniLMv2-L12-H384-v1, in GGUF

cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 in one f16 GGUF file, so llama-server can serve it as a /v1/rerank endpoint. Nothing about the model was changed: these are the published weights in a different container.

It exists because Semantic Deterministic Graph runs every model it uses on llama.cpp, and the published cross-encoder ships safetensors only. If you are using PyTorch, use the original: this file buys you nothing there.

The file


name	`mmarco-mminilmv2-l12-f16.gguf`
sha256	`1116037dfab848873f20248bb62df2090aec2e7fca03c99c27a5a0c8ce0f23b8`
size	242,628,704 bytes
precision	f16
converted with	`convert_hf_to_gguf.py`, llama.cpp

Why f16 rather than a quantization

A reranker used with a relevance floor is read for its raw score, not only for the order it produces: the floor is what turns "nothing here is about that" into an answer, and it is a number on this model's own scale. A quantization that preserves ranking can still move a score across a floor. f16 keeps the file small enough not to matter and the question closed.

Fidelity against the PyTorch model

Ten query/passage pairs scored both ways, GGUF against sentence_transformers.CrossEncoder on the original weights:

—ranking identical, Spearman 1.0000
—largest score difference 0.0124, on a range of about twenty

Serving it

bash

llama-server -m mmarco-mminilmv2-l12-f16.gguf \
  --reranking --pooling rank -c 4096 -ngl 99

--reranking with rank pooling is what turns a sequence-classification model into the /v1/rerank endpoint.

Two things will bite you if they are not said:

The model reads 512 tokens. A passage longer than that must be cut before it is sent, or the request comes back as a 500 rather than a score. Cut by tokens, through the server's own /tokenize, not by characters: the PyTorch model truncates silently at the same width, and cutting the same way is what makes the two comparable.

The context is shared across the server's slots. A question and one passage of a few hundred words take roughly three hundred tokens; -c 4096 leaves four slots a thousand each. Sized for one slot instead, a long passage is a 500.

What it was measured on

Not a public benchmark. These are 44 graded Italian questions over 182 notes, plus 10 unanswerable ones, from the project this conversion was made for. Same candidates, same session, the reranker as the only variable.

reranker	r@3	r@5	r@10	silent on 10 unanswerable	median latency
none (vector + lexical)	33/44	33/44	37/44	5/10	19 ms
Qwen3-Reranker-0.6B	38/44	41/44	42/44	6/10	18,340 ms
this model (PyTorch)	42/44	42/44	43/44	7/10	364 ms

Read it for what it is: one corpus, one language, one machine, questions written by the people who ran the test. It says this model beat the alternative here on every column, and it does not say what it will do on your corpus. Retrieval thresholds have never transferred between corpora in this project's experience, and neither does a recall number.

The base model's own card publishes no benchmark, which is why there is one here at all.

Attribution

—Model: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
—Training data: mMARCO, MS MARCO machine-translated into 14 languages
—Base: nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large

Apache 2.0, unchanged from the model this file was converted from.