pngwn/system-one-qwen3.5-4b-scorer

System One scorer — Qwen/Qwen3.5-4B-Base + scalar scoring head

A single-pass "System One" decision model in the shape of TypeSafe's Jev: it takes unstructured state plus a set of typed questions (yes/no, Choice, numeric Score) and returns a probability distribution over exactly the options the caller supplied — in one forward pass, with no autoregressive generation.

Each (state, question, option) triple is scored by a sequence-classification head and the per-question logits are softmaxed. Because the output space is the option set, the output is type-safe by construction rather than by post-hoc parsing: there is no token stream that can drift outside the schema.

Training code: `system_one.py` in this repository.

Intended use

Route, classify, prioritise or score a decision whose options you already know. This is not a chat model and not a generator — it cannot produce free text.

Results

Held-out test split, option sets uncapped at evaluation, temperature 1.75 fitted on val.

Validation split, all 9 families, per-task cap 64 (n=537): accuracy 0.752, ECE 0.032.

Calibration is the point

Temperature scaling cuts ECE from 0.135 to 0.044 — about 2× better calibrated than the prompted baseline, at identical accuracy. The raw head is overconfident; that gap is the whole reason a calibration stage exists.

The baseline row is not measured on the same rows: it covers 7 of 9 families (112 questions, 16 per family), because banking77 (77 options) and ticket routing (52) do not fit its 26-letter answer alphabet. On those 7 families this scorer averages 0.748 test accuracy against the baseline's 0.679.

Latency

112.3 ms per question at 4 options — one forward pass, option batch scored together. A smoke run measured 559.7 ms at 77 options. Latency scales with option count and sequence length, not with output length, because nothing is generated.

Training

• —data: pngwn/system-one-decisions — 12,913 train questions across 9 task families
• —2,200 optimizer steps (step-capped, ~1.4 epochs), batch 8 questions, max_len 384, option cap 16, lr 1e-4 cosine with 3% warmup
• —LoRA r=16 over all linear projections plus a new scalar score head (30.5M trainable of 4.24B)
• —bf16 with gradient checkpointing; a100-large; 2h58m wall clock including eval and push
• —train loss 1.682 (step 25) → 0.302 (step 2200); loss plateaued near 0.5 by step ~500

Limitations

• —Option-cardinality mismatch. High-cardinality tasks are trained with a cap of 16 options but evaluated over all of them (banking77 77, ticket routing 52), because the cap is what keeps batching tractable. tickets_queue is the visible casualty at 0.234 accuracy, and it is also the worst-calibrated task (ECE 0.244).
• —384-token truncation. Long states (MMLU questions, long reviews) are truncated, so MMLU and yelp_score are the weakest non-routing tasks.
• —Knowledge-heavy multiple choice is not the strength of encoder-style single-pass scoring. It trades world knowledge for latency and schema safety.
• —The ticket component of the training data is CC-BY-NC-4.0, so this model inherits a non-commercial restriction.
• —A 4-epoch run was attempted and cancelled around step 500: the measured 0.18–0.22 steps/s could not finish inside the timeout, and the script only pushes after training completes, so continuing would have produced nothing.
• —Accuracy is far below frontier models. The claims here are type-safe output, calibrated distributions and single-pass latency — not intelligence.