pngwn/system-one-qwen3.5-4b-scorer-smoke

System One scorer: Qwen/Qwen3.5-4B-Base + scalar scoring head

Scores each option of a typed question (yes/no, choice, score) in a single forward pass and softmaxes per question. No autoregressive generation; the output space is exactly the option set supplied by the caller.

Training code: system_one.py in this repository.

Training

• —data: pngwn/system-one-decisions (train questions 12913, epochs 2)
• —LoRA r=16, seq len 384, option cap 16, lr 0.0001
• —trainable params 30476800 of 4236230656; 40 steps in 153s
• —temperature 0.900 (fitted on val)

Test split (temperature from val)

Val split

Limitations

• —Trained with option sets capped at 16 for the high-cardinality tasks (banking77 77 options, ticket routing 52); evaluation scores all options.
• —Sequence length 384 truncates long states (MMLU questions and long reviews).
• —The ticket component of the training data is CC-BY-NC-4.0, so this model inherits a non-commercial restriction.
• —Accuracy is far below a frontier model; the claim here is calibrated distributions over a fixed option set at single-pass latency.