usejul/minicpm5-2b-decision

minicpm5-2b-decision (PyTorch, bf16) — v1.1

A decision model: one text (the state) and typed questions in, a probability per option out, in one forward pass. It never generates text. MiniCPM5-2B with a merged LoRA (r=16) and a pointer head, trained on multi-task decision data, in bfloat16, unquantized. The 4-bit MLX build is at `bdauzats/minicpm5-2b-decision-mlx-4bit`.

Made for [jul](https://github.com/bdauzats/jul) — local typed decisions with the interface of TypeSafe's (Jev) Python SDK, on Apple Silicon through MLX or anywhere else through PyTorch.

It follows the architecture and training code of Kev (Jared Palmer, Apache-2.0), which reconstructs a Jev-style model, applied here to a different base model and a different data mix.

What changed in v1.1

Trained on 44,720 records instead of 15,061, of which about 15% are French. French accuracy rises from 0.485 to 0.710 on MASSIVE, and English from 0.690 to 0.815 — the previous version's headline limitation, "another language costs about 20 points", is now about 10. Calibration error on 59-option questions falls from 0.241 to 0.069.

It is also trained for one epoch, not two. Keeping a checkpoint per epoch showed the second one buys 0.3 points of accuracy on unseen sources while more than doubling its confident errors (0.6% → 1.4%) and needing a much harsher temperature (1.29 → 2.14) that then misfits questions with many options. The first epoch is what ships.

The previous version stays available as revision="v1.0".

Use it with jul

jul models add minicpm5-2b-decision --repo bdauzats/minicpm5-2b-decision --backend torch
jul ask choice "Which team should handle this ticket?" -o billing -o shipping -o access \
    --state "I was charged twice for order 4411" --model minicpm5-2b-decision

from jul import TypeSafeClient, Choice, Noul, Score

client = TypeSafeClient(model="minicpm5-2b-decision")
answer = client.system_one(
    state={"subject": "Charged twice", "body": "I see two charges for order #4411."},
    questions={"team": Choice(instructions="Which team should handle this ticket?",
                              criteria={"billing": "Payments and refunds",
                                        "shipping": "Delivery problems",
                                        "access": "Login and account access"}),
               "angry": Noul(instructions="Is the customer angry?"),
               "priority": Score(instructions="How urgent is this ticket?",
                                 criteria=["low", "normal", "high"])})

jul needs no code specific to this model: the input format, the readout, the pointer head and its temperature are described in decision.json, next to the weights. Kev's own server also serves these weights, from the unquantized checkpoint.

Results

Four zero-shot classification sets (BTZSC: FinancialPhraseBank, Yahoo Topics, Empathetic, Massive), 200 examples each. Measured on the 4-bit MLX build, which is the protocol the v1.0 numbers were measured with, so the two columns compare to each other rather than to these unquantized weights — quantizing costs about 4 points.

On these English sets the two versions are level. The mean moves by 1.4 points where the sampling noise on 800 questions is about 1.7, and two sets go each way. What is not noise is Massive: +5.0 points of accuracy and a calibration error divided by 3.5 (0.241 → 0.069), which is where the gentler temperature of a one-epoch model shows.

Measured in PyTorch, fp32, before quantization, on sources the model was never trained on:

Per task on transfer-v4: contrastive authorization 1.00, SciQ 0.91, composition (or-not) 0.88, composition (and-or) 0.84, PAWS 0.81, QNLI 0.81, Emotion 0.75, TweetEval 0.69, composition (conditional) 0.56, MMLU 0.49, date arithmetic 0.38.

French

MASSIVE's French and English test sets are translations of one another: 200 utterances, 59 intents, options in English in both runs, so what is measured is whether it understands French, not French option names.

The gap is the same either way, so it is the model's and not the quantization's.

The v1.0 card said that in French jul's untrained vector method (0.535) beat this model. That is no longer true: 0.710 against 0.535. The remaining 10-point gap is the base model's, which loses about that much on the same pair read by the vector method.

Calibration

A temperature of 1.289, fitted once on the in-distribution development set, is applied at load time. It never changes an answer. On transfer-v4 it takes calibration error from 0.058 to 0.033 and confident errors — wrong with probability ≥ 0.9 — from 1.2% to 0.6%. Set KEV_TEMPERATURE=1.0, or pass your own, for the raw logits.

These are the unquantized weights, so the probabilities are the reference ones: the 4-bit build keeps the answer on almost every example but moves probabilities by up to ~0.23. If the probabilities themselves matter to you, fit a calibration on your own data (jul does this from a Context).

Latency

Measured on an M4 Pro (24 GB) on mains power, through jul, on the 4-bit MLX build — these bf16 weights are slower. A call encodes the state once; every question then continues from that cached prefix, so several questions about one text cost little more than the first.

Cost grows with the number and the length of the options, which are read on every request. jul can route questions with many options to its vector reading instead, which is flat in the option count — it measures the threshold and what it costs when you add the model.

Files

Training data

Kev's decision-v7 suite (12,576 records), 2,800 of our own English sets (dair-ai/emotion 1,000, and 300 each of CAP SOTU, Manifesto, app reviews, Bias Frames intent, Wikipedia toxicity, TrueTeacher), and 7,299 French/English records built from MTOP intent and domain, Amazon reviews (fr), XNLI and PAWS-X — French questions and options, plus about 20% English-option twins. 44,720 records in all, none rejected or truncated.

The public sources include AG News, Banking77, DBpedia14, TREC, SST-5, Yelp, Amazon reviews, IMDB, BoolQ and MNLI, each from its training split, with its own license. Results on those tasks are therefore not zero-shot, which matters if you compare this model with one that has never seen them.

One epoch, LoRA r=16 α=32 on q/k/v/o_proj and gate/up/down_proj, pointer head from scratch, cross-entropy, lr 5e-5 (one-cycle), batch 8, bf16 autocast with fp32 master weights; 2 h 34 on one L40S.

Limitations

• —The Jev benchmark has not been read for this checkpoint. v1.0's numbers on it do not carry over.
• —French is measured on one dataset. 0.710 against 0.815 in English is a real gap, and nothing here says how it behaves in a third language.
• —A state longer than 384 tokens, or a question longer than 1024, goes beyond what it saw in training.
• —Weak on knowledge and arithmetic: on transfer-v4, MMLU 0.49 and date arithmetic 0.38. It sorts and decides; it does not reason over facts it was never given.
• —No evaluation outside the sets above, and none of them is a benchmark of your own data. Measure it.