jbarney/circuit-8b

circuit-8b

v1.1 (2026-09-21). Same base, head and recipe as v1.0, with two question families added to the training data: is this claim supported by the evidence, and does this tool fit the request. It gives back a point or two on the tasks v1.0 was tuned on and gains 5 to 15 on tasks neither had seen. See Versions. revision="v1.0" gets the original weights.

A System One decision model: typed questions in, calibrated probability distributions out, one forward pass, no text generation. It speaks TypeSafe's POST /v1/systemone contract and is the larger open-weights backend for decision-circuits. The small one is circuit-1.7b.

circuit-8b is a LoRA adapter plus a pointer readout head on Qwen/Qwen3-8B-Base. A request packs a state and a question; each option is wrapped in delimiter tokens and the sequence ends with a decide token. The head scores every option's closing delimiter against the decide token and applies softmax. Those probabilities are the answer, trained with cross-entropy against outcome labels, so calibration is learned rather than derived.

Results

Same items for every model, human labels, accuracy / ECE (15 bins).

The four cold-eval tasks' train splits are in this model's training mix; they are presumably not in Jev's. Checks nothing was trained on:

\* the grid generator is ours, so this is held-out items, not held-out structure. With the 1.7B recipe, an operation and a format withheld scores 91% on the unseen format and 57% on the unseen operation: layouts transfer, new kinds of judgment must be in the training data.

Known weakness shared with every model above, Jev included: on inputs built to be undecidable, mean confidence is 0.5 to 0.85 where it should be near 0. Calibration on ambiguity is the open problem.

Throughput on one RTX A6000 (48 GB), bf16, batch 8: 72 ms per grid item, about 2,100 input tokens per second. On the 546 production questions (longer states) it is 198 ms per item.

Datasets nobody prepared for

300 items each from four public datasets that are in no model's training mix here. Accuracy / ECE / Brier score; lower is better for the last two. ChaosNLI's reference is the vote of 100 annotators per item, so its Brier measures distance from how people answered. (An earlier version of this card reported KL. Jev's API rounds to two decimals, which makes KL against it meaningless, so this table uses Brier.)

Reproduce all of it with one command: REPRODUCE.md. The ChaosNLI gain is mostly the checkpoint, not the new data: the last checkpoint of the same run scores .577 with KL 1.11. Training on one-hot labels past the point of best validation calibration buys accuracy where the training data lives and spends calibration everywhere else, so v1.1 is the early checkpoint on purpose.

Training

• —Base: Qwen/Qwen3-8B-Base (Apache 2.0), frozen. LoRA rank 16, alpha 32, on all attention and MLP projections (43.7M params). Pointer head: two 4096 x 256 linear maps.
• —Data, 19,738 items, the same mix as circuit-1.7b: every label computed by code or by humans, every source permissive (CC BY / CC0 / MIT / Apache). The generalization grid (8,100 + 838 oversampled ambiguous items with soft 0.5 labels), a commercial-safe slice of public classification tasks (2,100), and CC-licensed real data (MNLI, civilcomments, smsspam, CLINC; 5,700). New in v1.1, 3,000 items: claim and answer support from VitaminC and SQuAD v2, and tool relevance, tool choice and tool-call checks built from CLINC and SNIPS utterances against hand-written tool specs. BFCL and HaluEval supply nothing; they are the test. No teacher-model outputs.
• —1 epoch, batch 4, max 1,024 tokens, lr 1e-4 (LoRA) / 1e-3 (head), bf16 with gradient checkpointing, soft-target cross-entropy, early stopping on validation ECE (kept step 1,200 of 4,446: ECE 0.020, accuracy 89.9%). 45 minutes on one H100. v1.0 was step 2,800 of 3,766, 75 minutes on one RTX A6000.
• —Training code, data generators, and the evaluation harness: github.com/Barneyjm/circuit.

Versions

Pin one with revision="v1.0" in huggingface_hub, or hf download jbarney/circuit-8b --revision v1.0.

Use

Serve it with the circuit repo and point any System One client at it:

S1_MODEL=lora:runs/circuit-8b uv run python -m s1proto        # :8901

from decision_circuits import Circuit, Q, argmax
from decision_circuits.backends import SystemOne

c = Circuit()
c.noul("pii", "Does the message contain personal information?")
c.choice("dept", "Which team?", {"billing": None, "technical": None, "other": None})
c.gate("redact", Q("pii") >= 0.7, on_uncertain="escalate")
c.gate("route", argmax("dept", min_confidence=0.35))
out = c.run(SystemOne("http://localhost:8901/v1/systemone", api_key="x"), "Card charged twice, my card ends in 4412")

Files: adapter/ (PEFT LoRA), head.pt (pointer head, keys q.weight, k.weight), config.json (base, hidden size, head type, layout, training args). The base needs about 17 GB in bf16; set "load_4bit": true in config.json and the server loads it in 4-bit (bitsandbytes NF4), which fits a 12 GB card.

Intended use and limits

Research and evaluation of calibrated decision models, and as a local backend for decision circuits. Not a production system for decisions that affect people. English only. Confidence on undecidable inputs is not yet reliable (see above); put thresholds inside an uncertainty band and route the band to a human, which is what decision-circuits does.

License

Adapter and head: Apache 2.0. Base model: Qwen license (Apache 2.0 for Qwen3).