ZY-Lee4/QJev3.5-0.8B-GGUF

QJev 3.5-0.8B: a small decision engine you can run on a laptop

https://huggingface.co/ZY-Lee4/QJev3.5-0.8B-GGUF/resolve/main/qjev_demo.mp4

Fifteen seconds of the shipped adapter deciding, end to end: permission review on real agent commands, context compaction over a live session, browser-action scoring, and the official System One HTTP schema, with the probabilities on screen. Nothing in the clip is scripted; it is one run of the project's demo script, recorded automatically.

This is a 0.8B model we fine-tuned to make one narrow kind of decision very cheaply. It reads a state, a question, and a list of options, then returns a probability for each option. It never writes text. One forward pass, one token, and you have a distribution your code can branch on.

We built it for our own product first. An educational chatbot needed to decide, before spending a full LLM call, whether a student message was on topic, whether a tool call was safe to run, and whether an old tool result still had to be kept verbatim in the context. Those are small bounded judgments, and paying a 70B model to make them was absurd. It runs today on an office GPU at 80 to 150 ms per decision, and on a laptop with llama.cpp.

Files

Fidelity was measured on our 180 question product set: the merged models agree with the base-GGUF-plus---lora path on 179 of 180 (f16) and 178 of 180 (Q8_0) decisions, at 94.4% and 93.9% accuracy. Quantisation costs one or two answers in 180, nothing more.

One thing to know about that 180: 29% of its rows appear verbatim in the training mix (38 of its 100 gate rows, 15 of its 30 permission rows; its email and compaction rows are clean). The agreement figures above are unaffected, since they compare two ways of serving the same weights, but the accuracy on that set is partly a memory score. The probes we trust for generalisation are the extended email and compaction sets, which do not overlap training at all (88.2% and 92.9% for the released adapter), and the external benchmarks in the table below.

release/fetch_model.sh in the project downloads the base GGUF plus the adapter GGUF and checks both hashes, so nothing here needs a full model to be repackaged.

What it does well, and what it does not

We ran the whole thing against the official Jev 1.13 API, against three open competitors, and against the base model without any fine-tuning. The short version is in the chart below.

[image]

Left two bars: our own task distribution, the one the model was trained for. Right two: held-out general benchmarks, where the base model is as good as our fine-tune.

On the distribution it was built for, fine-tuning is worth +67 points over the untouched base model (27.2% to 94.4% on 180 questions). On held-out general benchmarks it is worth nothing: the base model scores 50.6% on jev-bench held-out configs and our fine-tune scores 49.3%, with overlapping confidence intervals. The general axes belong to the backbone, and at 0.8B the backbone sits near 50%. The official API reaches 75.0% there. If general decision accuracy is what you need, use a larger model; the 4B prompt-based baselines land at 74 to 79% on JevBench.

We would rather write this down than let you find out after deploying.

[image]

Each point is a version we trained, scored on 231 public tasks (left) and 960 jev-bench rows (right). The solid line counts every source. The dashed line counts only sources the model never trained on. The two lines tell different stories about the same nine runs.

That gap is the most useful thing we learned. We spent two days improving a number that turned out to be mostly memorisation of the sources we trained on. After rebuilding the evaluation to hold out sources rather than records, the honest picture is: v1 to v11 gained 5 points on jev-bench held-out configs (44.3% to 49.3%) and 2.9 points on held-out JevBench families, where the confidence interval is 9 points wide. The rest was in-distribution.

Product family readout

The work we care about happens in four families: routing a message, judging a tool call, keeping or dropping context, and sorting an email. Each is measured on its own held-out questions. We extended the email and compaction sets past 100 questions each after discovering that 30 questions gave us a confidence interval 23 points wide.

[image]

Permissions and gate on the left, email and compaction on the right, both above 100 questions. Email is the hardest family, and the one the v14 data work targeted.

The four families are not equally hard, and that spread matters more than the average. On the extended probes gate and permissions sit at 97 to 100% and email and compaction at 74 to 79%. Email moved least across our first twelve training runs, which is what the v14 data work targeted; the released adapter now reaches 93.3% on the product email probe and 88.2% on the harder extended one.

One honest note about versions, and it cuts both ways. The released adapter is v14s0, which beat the previous release (v5) on every product instrument in a ten-battery run on identical instruments: the frozen 180 question set 94.4% against 93.9% (one flipped answer against four), the 222 extended email/compaction probes 88.2% and 92.9% against 69.1% and 75.9%, permission review on real commands 87.2% against 58.1%, gates unchanged at 100%. Most of that gain turned out to be a fix rather than more knowledge: v5 answered permission questions at 80% in the option order it was trained on and 46.7% with the options reversed, and per-row option shuffling in v14s0 removed that swing. The same shuffling is why v14_s0 handles the three way NLI format at 89.3% where v5 managed 41.3%.

The part that does not cut our way: that training recipe is not reliable. Four seeds of it produced two strong adapters, one that collapsed halfway and one that collapsed to the base model, and the training loss ranks the outcome almost perfectly, so any future retrain has to gate on it. v14s0 is shipped as a validated artefact, not as a reproducible procedure. On the general reasoning axis nothing here beats the base model's ceiling either: held-out sources score the same before and after fine-tuning (about 50%), and JevBench lands at 57.6% for v14s0 against 61.9% for v5, both far behind the official System One implementation. Every number in the tables above says which version it came from.

Latency, and how the alternatives compare

[image]

One protocol, p50 per decision. The official API number includes the network round trip; the local competitors were measured on their own recommended hardware.

Measured by us, each engine using its own prompt contract:

We are ahead where we trained and behind where we did not. For a model that has to fit in 812 MB and answer in under a tenth of a second, that trade seems right.

Reliability

[image]

149 of 180 questions land in the top confidence bin, and they are right 96% of the time. ECE 0.0427. Confidence follows the official Jev formula, `c = (pmax - 1/K) / (1 - 1/K)`, computed from the option letter distribution.

Confidence here is a statistic of the distribution, not a claim that the answer is correct. Use it to route: act above a threshold, escalate below. Out of domain it is over-confident, and so is the official API. On the three jev-bench configs where human annotators disagree with each other (ChaosNLI, hate speech vote shares, Civil Comments), no model we measured tracks human uncertainty, the official one included. We report total variation distance to the human distribution rather than pretending that axis is solved.

Quick start

Pair the adapter with the base model, which is the smallest download:

llama-server -m Qwen3.5-0.8B-Q8_0.gguf --lora qjev35-lora-v14_s0.gguf \
  --port 8080 -c 4096 --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

python qjev_client.py --url http://localhost:8080 \
  --state "The build failed: missing pyarrow in the test venv." \
  --question "What should the agent do next?" \
  --options "install the dependency" "retry the build" "ask the user"

Or run a merged file on its own:

llama-server -m qjev35-0.8b-q8_0.gguf --port 8080 -c 4096 --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

In LM Studio: download either merged GGUF, load it, start the local server from the Developer tab, and point qjev_client.py at http://localhost:1234 with the model name LM Studio shows.

qjev_client.py uses nothing but the standard library. It asks for one token with top_logprobs from /v1/chat/completions and renormalises the option letters:

from qjev_client import decide, decide_bool, decide_score

decide("http://localhost:8080",
       "Disk usage hit 96% on the build host.",
       "Should the agent stop the pipeline?",
       ["stop and clean up", "continue", "escalate to a human"])
# [('stop and clean up', 0.91), ('escalate to a human', 0.07), ('continue', 0.02)]

decide_bool("http://localhost:8080", evidence, "Does the passage entail the claim?")  # P(true)
decide_score("http://localhost:8080", ticket, "How urgent is this?", levels)         # distribution

Option order does not matter. The client evaluates the order you gave and the reverse order, then averages the two distributions. That trick is worth 2 to 4 accuracy points at this model size, and we have used it since the first version.

Prompt format

Set the chat template with thinking disabled. The model was trained on this shape:

system: You are a judgment engine. Read the question and the options, then decide which single
        option best matches the state. Answer with EXACTLY ONE letter and nothing else.

user:   [QUESTION]
        {question}

        [OPTIONS]
        A. {option 1}
        B. {option 2}

        [STATE]
        {state}

The next token is the option letter. Its probability, renormalised over the letters you declared, is your distribution. For yes/no questions pass two options (false, true). For ordered levels, list them in order. The released GGUF ships a pass-through chat template, so a chat endpoint hands this text to the model unchanged.

Read the letters from log probs. Sampling text and parsing it throws away the whole point.

Scope

Good for bounded decisions where you know the options when you call the model: tool gating, permission allow/ask/deny, prompt injection and content checks, routing, context compaction, email triage, picking an entity, browser actions, and four ordered score dimensions.

Not for open-ended reasoning, general knowledge, mathematics, multi-hop planning, long document question answering, or judging free-form assistant output. The held-out numbers above are the reason, and they are properties of the 0.8B backbone.

Option cap is 26, using positional letters A to Z. The official Jev allows 255. That is a design limit rather than an oversight. Two separate reasons stop us running 77 way (Banking77) or 151 way (clinc150): the letter readout caps at 26 options, and, measured after this card was first written, the adapter answers a 25-option Banking77 question drawn from the full label space at 3.9% (154 rows, chance is 4.0%). Our earlier Banking77 numbers came from rows whose distractors were a small sampled subset, and most of those rows sat in the training mix, so they measured memory tasks. We report it as a limitation instead of working around it with a hack.

Training data and license

Seventeen families, all synthetic or converted from public datasets: gate, permissions, compaction, email, entity, browser actions, four score dimensions, NLI (SciTail, MNLI, ANLI), CLINC, Banking77, MASSIVE Chinese, ToolACE, WANLI, CMNLI, tldr-pages, and nl2bash. Per-source licences are listed in SOURCES.md in the dataset repository. ANLI (cc-by-nc-4.0) is used for internal training only and is stripped from every published export. Weights are Apache-2.0, following Qwen/Qwen3.5-0.8B.

How we evaluate

Five rules, written down after we broke them:

1. 1.Hold out sources, not records.
2. 2.Treat any difference smaller than its confidence interval as no evidence.
3. 3.Fit on validation, report on test.
4. 4.Require at least three training seeds before claiming one version beats another.
5. 5.Keep domain numbers, held-out numbers, and competitor numbers in separate columns.

They are the reason the tables above are less flattering than our first release notes, and the reason we trust them more.

Known limitations

• —Held-out general accuracy sits near 50%, which is where the 0.8B backbone sits.
• —Five and six level ordinal scales that we did not train on score near chance, 25 to 27%.
• —Confidence is over-confident out of domain, on our model and on the official API alike.
• —The model leans on canonical option order more than one trained on shuffled orders would. Keep permutation averaging on, and if you train on this data yourself, shuffle the options per row. We found a fixed option order baked into 1,953 rows of our own training data; fixing it moved the position uniformity statistic from z = +6.6 to z = -0.2.

Citation

@misc{qjev35,
  title  = {QJev 3.5-0.8B: a small decision engine},
  author = {Lee, Jerry and contributors},
  year   = {2026},
  url    = {https://huggingface.co/ZY-Lee4/QJev3.5-0.8B-GGUF}
}

Independent work, written from scratch against the publicly described System One pattern. Not affiliated with, endorsed by, or connected to TypeSafe AI.