cosetoenor/decider-2b-GGUF

[中文介紹與 v10 更新說明](README.zh-TW.md) · [v8 中文量化測試報告](BENCHMARK_3070TI_STATE_FIRST.zh-TW.md) · [中文三模型比較](BENCHMARK_THREE_MODELS.zh-TW.md)

decider-2b-GGUF

GGUF conversions of Mapika/decider-2b (v10) for llama.cpp. Decider-2B is a fine-tune of Qwen3.5-2B-Base. It reads a state and typed questions with described options, and returns option probabilities from one forward pass. It does not generate text.

All credit for the model goes to its author. This repo only changes the file format. See Changes.

v10 release (2026-09-20)

Weights updated to v10. Previous evaluations primarily cover v8. For upstream changes, training details, and evaluation protocols, refer to the [original repository](https://github.com/Mapika/decider) and its [RL report](https://github.com/Mapika/decider/blob/main/docs/RL.md).

The root GGUF files now use upstream v10, pinned to 4a0e86782adfdb7393e04b8ec9f6b939dca09273. The previous files, reports, and code remain available at revision v8. See RELEASE_V10.json for source hashes, conversion provenance, and validation scope.

This is an optional upgrade, not a demonstrated improvement on every task. Upstream reports:

Source: upstream evaluation. These are upstream model results, not measurements of these GGUF files. Browser evaluation covers synthetic click tasks, not real-site typing or scrolling. The architecture is unchanged; updating weights alone does not establish faster inference.

Files (v10)

Q8_0 is the conservative starting choice based on the v8 evidence below. The v10 files have conversion and structural validation only; runtime inference and accuracy have not been validated. The bundled server retains the schema-first packed-question behavior. Its v10 runtime compatibility is untested.

Historical v8 quantization agreement

All measurements below, including GPU, three-model, and OCI CPU results, use v8. They do not describe v10.

How these numbers were measured: 71 real multi-turn customer-service routing prompts (mostly Traditional Chinese), 2 to 10 scenario options each, questions-first layout, all questions packed in one row. The reference is the original bf16 safetensors run through the author's decider package in PyTorch. "Picks equal" compares the chosen option. It is agreement with the original model, not accuracy against human labels.

Historical v8 GPU benchmark: state-first prompts

On an RTX 3070 Ti with 8 GB VRAM, Q80 and IQ4NL each completed 7,440 evaluation cases using the same llama.cpp engine, prompts, and configuration.

This benchmark uses the author's default state-first layout: state → question/options → answer slot. The 71-case routing test above and the bundled caching server use schema-first: question/options → state → answer slot. They are separate evaluations. The BF16 reference uses PyTorch, so comparisons with GGUF include engine and precision differences. CLINC configurations reuse utterances; every gold-removed case should select None of the above. Latency excludes model loading and HTTP.

Full GGUF report: English · Traditional Chinese · Data

Historical v8 three-model comparison

NanoJev, Decider-2B, and System One Scorer were evaluated on the same 7,440 cases per model, for 22,320 predictions. Models ran sequentially on the same GPU. The report includes correct counts, accuracy, P50/P95 latency, dataset and model sources, and downloadable predictions.

This compares deployed configurations, not architectures at equal precision: NanoJev is the maze-specialized local_atomic_seed17 checkpoint; Scorer uses NF4 to fit the 8 GB GPU; Decider uses BF16. In this deployment, Decider offers the strongest latency/accuracy balance, while Scorer is more accurate on BoolQ and TMMLU+ but slower.

Three-model report: English · Traditional Chinese · JSON · CSV

Use

A plain llama-cli or llama-server run gives a text model that continues a prompt. To get Decider's option probabilities, build Decider's prompt, read the logits at each answer slot, restrict them to the option-letter tokens, and apply softmax. code/ does this.

• —code/dz_shim.c: a small C wrapper over libllama. It decodes tokens, returns the logits of chosen token ids at chosen positions, and saves or restores the state of sequence 0 (attention KV plus the Gated DeltaNet recurrent state).
• —code/decider_llama_serve.py: a POST /v1/systemone server (the wire format of the author's decider.serve). It decodes the question prefix once per question set, snapshots the sequence state, and for each request restores the snapshot and decodes only the state. This is the author's questions-first (schema cache) layout.

The server imports the prompt builder and the tokenizer from a local copy of the original repo, so download that too.

huggingface-cli download Mapika/decider-2b --revision 4a0e86782adfdb7393e04b8ec9f6b939dca09273 --local-dir decider-2b --exclude "*.safetensors"
huggingface-cli download cosetoenor/decider-2b-GGUF decider-2b-q8_0.gguf code/dz_shim.c code/decider_llama_serve.py --local-dir .

# llama.cpp as a shared library (CPU build shown)
git clone https://github.com/ggml-org/llama.cpp && cmake -S llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=ON -DGGML_NATIVE=ON
cmake --build llama.cpp/build -j --target llama
gcc -O2 -shared -fPIC -o libdz.so code/dz_shim.c -Illama.cpp/include -Illama.cpp/ggml/include \
    -Lllama.cpp/build/bin -lllama -Wl,-rpath,$PWD/llama.cpp/build/bin

pip install fastapi uvicorn transformers
cd code && DECIDER_MODEL=../decider-2b DECIDER_GGUF=../decider-2b-q8_0.gguf DECIDER_SHIM=../libdz.so \
    uvicorn decider_llama_serve:app --port 8000

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"latest": ["I was charged twice for the same order"]},
  "questions": {"team": {"type": "choice", "instructions": "Which team should handle `latest`?",
                          "criteria": {"billing": "Charges and invoices", "shipping": "Delivery status", "other": null}}}}'

Environment variables: DECIDER_THREADS (default 4), DECIDER_BATCH (default 512), DECIDER_MMAP (default 0), DECIDER_MAX_SCHEMAS (cached question sets, default 16). One server serves one request at a time.

Historical v8 performance notes

Measured on an Oracle Cloud A1 VM (Arm Neoverse-N1, 4 threads, dotprod but no i8mm or bf16), Q80, `DECIDERBATCH=512, DECIDER_MMAP=0`:

On Arm, llama.cpp repacks Q80 weights for its dot-product kernels (`CPUREPACK` in the load log). With mmap off, the repacked copy is the only copy in memory. IQ4NL was not measured on Arm. On this CPU Q80 was about 2x faster than F16.

Cost per request grows with the state, so keep the state short, and decode each question set before traffic arrives.

Caveats

• —The questions-first layout costs accuracy. The author reports held-out accuracy 0.741 for the default state-first layout and 0.707 for questions-first (temperature 1.30 vs 1.18). The shipped server uses questions-first because it is the layout that can be cached.
• —The author lists weak held-out tasks (instruction categories, preference judgements, long-document reading). Check the model on your own labelled data before you rely on it.

Changes

• —v10 converted with llama.cpp 4fea119de30f6a923992780f6fd5ccb0bee5d47d on 2026-09-20. Q80 was converted directly from bf16. IQ4NL was quantized from the F16 file with llama-quantize.
• —--no-mtp was used. The original config.json declares mtp_num_hidden_layers: 1, but the checkpoint has no MTP weights. Without the flag, the GGUF declares 25 blocks and llama.cpp fails with tensor 'blk.24.attn_norm.weight' not found.
• —No weights were retrained or merged.

License

Apache-2.0, the same as Mapika/decider-2b and Qwen/Qwen3.5-2B-Base. See LICENSE.