bdauzats/WeMM-Embedding-4B-mlx-4bit

WeMM-Embedding-4B — MLX 4-bit, text only

A 4-bit MLX conversion of `tencent/WeMM-Embedding-4B`, for text embeddings on Apple Silicon. 2.6 GB instead of 10.3 GB, same accuracy as the bf16 original on our measurements.

Who made what. The model is Tencent's (WeMM team, technical report): a multimodal embedding model built on Qwen3.5-4B, released under Apache-2.0. This repository only converts it, for [jul](https://github.com/bdauzats/jul) — local typed decisions (Choice, Noul, Score) with calibrated probabilities and the interface of TypeSafe's (Jev) Python SDK, on Apple Silicon. Nothing was retrained.

What changed from the original

• —Text only. The vision tower is dropped by the conversion; images and video are not supported here. Use the original for them.
• —4-bit affine quantization, group size 64 (4.5 bits per weight), with mlx_lm.convert 0.31.3: mlx_lm.convert --hf-path tencent/WeMM-Embedding-4B -q --q-bits 4 --q-group-size 64.
• —The Sentence Transformers and SGLang helper files are removed (they need the PyTorch model).

Embeddings stay close to the bf16 original: cosine 0.90 to 0.98 between the two versions on the same text, and the same similarity structure. On classification the difference is nil (below).

What it is for — and what it is not

Use it to sort a single text into labels you describe in words, with no training data: routing a ticket, tagging a topic, an intent, a sentiment or an emotion. Embed the text and each label once, compare by cosine, and a single temperature turns the scores into calibrated probabilities. It is the best zero-shot text classifier we have measured, and it reads French as well as English.

Do not use it to decide anything that requires reading two things together: whether two sentences mean the same thing, whether a case satisfies a policy, whether a report is late. An embedding model encodes the text and each option separately, so it cannot compare them. On several of those tasks it scored below always answering the majority class. A model trained to decide (a cross-encoder, or `bdauzats/minicpm5-2b-decision`) is the right tool.

Usage

import mlx.core as mx
from mlx_lm import load

model, tokenizer = load("bdauzats/WeMM-Embedding-4B-mlx-4bit")

def embed(text: str) -> mx.array:
    chat = [{"role": "user", "content": [{"type": "text", "text": text}]}]
    prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=False)
    ids = tokenizer._tokenizer(prompt)["input_ids"]          # appends the <embedding> token
    h = model.language_model.model(mx.array([ids]))[0, -1]   # hidden state at <embedding>
    return h / mx.linalg.norm(h)

labels = ["billing: payments, invoices, refunds", "technical: bugs, errors, crashes", "sales: pricing, plans, demos"]
L = mx.stack([embed(l) for l in labels])                     # once per label set
q = embed("Which team should handle this ticket?\nI was charged twice for my subscription this month.")
probs = mx.softmax((L @ q) / 0.0219)
# {'billing': 0.875, 'technical': 0.022, 'sales': 0.104}

• —Query = the question, a newline, then the text. Labels = plain text; a short description helps (full sentences scored 6 points higher than bare names on our development sets).
• —Temperature 0.0219, fitted by log-loss on the four development sets below. Raw cosines have no probabilistic meaning, so always apply one. Fitted on three sets and scored on the fourth, it stays between 0.021 and 0.023: a single value transfers across tasks. Refit it on your own labeled data if you have some.
• —Pooling and chat format are those of the original model card: last token (<embedding>), L2-normalized, 2 560 dimensions. Matryoshka truncation is not measured here.

Measurements

All on an M4 Pro (24 GB), MLX, one query at a time with label embeddings cached. ECE uses the temperature fitted on the other development sets, never the set being scored.

Development sets (BTZSC, 200 examples each, label sentences, zero-shot):

Mean 95 % interval: 0.654–0.716. 4-bit against bf16, paired on the same examples: +0.002 (−0.016 … +0.021).

French: MASSIVE's parallel English and French splits, 0.595 and 0.590 — no language gap.

Jev benchmark (AbdelStark/jev-benchmarks, BTZSC pilot v1, 300 rows, run once with the settings above). Cells: accuracy / ECE.

Read with care: Banking77 and Emotion are part of MTEB, which embedding models are trained and tuned on, so this model has probably seen them. Emotion at 0.78 zero-shot, where every other zero-shot model scores 0.46–0.48, points that way. On AG News, which is not in MTEB, it is 7 points behind Jev.

Decision tasks (Kev's transfer-v4, 656 questions): 0.643, against 0.739 for minicpm5-2b-decision and 0.857 for Jev. Good on single-text tasks (SciQ 0.96, Emotion 0.75, offensive tweets 0.71), below the majority class on paraphrase (PAWS 0.45 against 0.56), two of three policy compositions (0.44 and 0.47 against 0.50 and 0.56) and deadlines (0.28 against 0.50).

Latency depends on the text length: 52 ms for a short utterance, about 240 ms for a paragraph.

License

Apache-2.0, as the original; see LICENSE (Tencent's, unchanged). This is a modified version: quantized to 4 bits, vision tower removed.