oraculumai/Manchego-MLX-8bit

<p align="center"><img src="manchego.jpg" alt="The Manchego mascot: a smiling wedge of Manchego cheese in a black beret, waving" width="420"></p>

Manchego — MLX 8-bit

How Manchego v2.1 compares with every trained open ~4B model (28, each run as shipped on identical rows): 3= of 28 on JevBench's 231 public decisions, 5th on 27 held-out task types (none resolved ahead), 15th on Public8, where every model resolved ahead trained on, or cannot be cleared of, Public8's source data. Caveats and the full comparison: the main card. Three caveats apply, as on the main card: our training families were designed from an earlier version's per-family JevBench hard-tier scores (the published spec only, no item text); an early audit used Jev (TypeSafe AI's hosted decision service) to flag phrases in two of our data pools (86 phrases dropped); and the first stage's registered checkpoint selection read one development group whose targets came from Jev (a rule excluding that group selects the same checkpoint). This repository holds v2.1 only (tag v2.1); earlier versions are not published.

[image]

Manchego v2.1 converted with mlx_lm at 8 bits (group size 64; 4.5 GB) for Apple Silicon. A 4B decision model: state + question + options in, a probability per option out, one forward pass, no generated text.

Accuracy with cross-entropy in brackets, short prompt, one prompt per forward pass. Within one decision of the bf16 weights on the 80 task rows, one better on the real text; cross-entropy within 0.003.

pip install "mlx-lm>=0.31.3" (tested with 0.31.3). Text-only conversion: the vision tower and the multi-token-prediction head are dropped, so load it with mlx_lm, not mlx-vlm. English.

from mlx_lm import load
import mlx.core as mx
model, tok = load("oraculumai/Manchego-MLX-8bit")
prompt = "Is this message spam?\n\nState:\nWIN a free cruise! Reply YES now.\n\nOptions:\nA = yes\nB = no\n\nReply with only the letter of the best option."
letters = "AB"                                                        # the letters you offered
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
logits = model(mx.array([tok.encode(text, add_special_tokens=False)]))[0, -1]
cand = [tok.encode(L, add_special_tokens=False)[0] for L in letters]
probs = mx.softmax(logits[mx.array(cand)].astype(mx.float32))
print(dict(zip(letters, probs.tolist())))

Scoring several prompts as one padded batch moves probabilities slightly (measured on this conversion; 164 questions: up to 0.055 with one changed decision under the short prompt, up to 0.090 with none under the state-first prompt); score one prompt at a time when exact reproducibility matters.

The short prompt, fixed (use it for up to 26 options; for larger menus or untrusted state use the state-first prompt, contract_v2.py in the main repository):

<question>

State:
<state text or JSON>

Options:
A = <value> — <description>
B = <value> — <description>

Reply with only the letter of the best option.

One user message, chat template, thinking disabled, generation prompt added. Read the distribution over the letters you offered at the first generated position.

Read the main card first

What the model is, how it was trained, every result and every weakness are on the main card: `oraculumai/Manchego`. The most important ones: v2.1 repairs v2's overconfidence on unfamiliar task definitions (confirmed once on a sealed set) but is not more accurate than v2; on the sealed set of 25 unfamiliar tasks it is not more accurate than the untrained base (on 27 source-clean held-out tasks it is +0.024 above it, a gain from the first stage); on JevBench its accuracy is not separated from v2's; it costs 1.6 to 2.6 points on two of the project's own policy-document tests. With two passes to spare, average the short prompt's and the state-first prompt's distributions (details, *How to use it*).

Training data attribution and licences (WANLI, MultiNLI, Bitext customer support, banking77, the 42 upstream datasets of the Super-NaturalInstructions tasks, ViZDoom/Freedoom) are on the main card and in its NATURAL_TASKS_ATTRIBUTION.md. Illustration: the Manchego mascot, an AI-generated image (ChatGPT image generation) supplied by the project's author.