crh225/plumb-4b-GGUF

Plumb-4B GGUF

Plumb-4B for llama.cpp and Ollama, on almost any GPU or a CPU.

Plumb-4B is a decision model, not a chat model: it answers with one option letter, and its value is in the probabilities of those letters, softened by its calibration temperature T = 2.07.

With Ollama

Modelfile in this repo holds the decision prompt as the template, so an ordinary chat request carrying the question as JSON (evidence, criterion, lettered options) renders the right prompt:

ollama create plumb-4b -f Modelfile      # next to the downloaded .gguf

Ask for one token with logprobs and top_logprobs, send reasoning_effort: "none", read the option letters' log-probabilities, divide by 2.07 and renormalise.

With llama.cpp

llama-server -m plumb-4b-v5-Q8_0.gguf -c 8192 -ngl 99
pip install --no-deps "jevk5 @ git+https://github.com/allebee/jevk5@v0.2.1"   # standard library only

from jevk5 import JevK5GGUF

model = JevK5GGUF(temperature=2.07)   # llama-server on :8080
model.decide("Order #7120 shows delivered to No. 17; the customer lives at No. 71.",
             {"type": "choice", "instructions": "What happened to the parcel?",
              "criteria": ["delivered", "misdelivered", "unknown"]})

Converted with llama.cpp (convert_hf_to_gguf.py --no-mtp, then llama-quantize). Apache-2.0; see the main model's card for credits.