LocalLLaMA/typed-decisions
Typed Decisions A benchmark for typed probabilistic decisions over shared state. You give a model one piece of unstructured state. It answers several typed questions about that state at once, and every answer is a probability distribution rather than a single label. The schema follows the System One primitives used by TypeSafe AI: noul, choice and score. A row replays against any API that implements that shape. This benchmark is independent. It is not affiliated with TypeSafe… See the full description on the dataset page: https://huggingface.co/datasets/LocalLLaMA/typed-decisions.
Typed Decisions
A benchmark for typed probabilistic decisions over shared state. You give a model one piece of unstructured state. It answers several typed questions about that state at once, and every answer is a probability distribution rather than a single label.
The schema follows the System One primitives used by TypeSafe AI: noul, choice and score. A row replays against any API that implements that shape. This benchmark is independent. It is not affiliated with TypeSafe and it does not reproduce their Jev model.
What it is for
One question sits behind the dataset. Does breaking a workflow into typed probabilistic decisions, especially with a shared encoder, buy you a better accuracy / calibration / latency trade-off than direct classification or than prompting an LLM?
Answering that needs several typed questions over one input, answers that are genuinely probabilistic, and latency you can measure per decision. That is what this is.
The three question types
Every option carries a written description in criteria. Those descriptions are part of the input. Strip them to a bare label list and you have a different, easier task.
Workflows
Each case asks 5 questions over one shared state.
Columns
state and questions together are exactly the body of a POST /v1/systemone request. You can replay a row without reshaping it.
Usage
from datasets import load_dataset
import json
ds = load_dataset("LocalLLaMA/typed-decisions", "customer_service", split="test") # benchmark
tr = load_dataset("LocalLLaMA/typed-decisions", "customer_service", split="train") # training data
row = ds[0]
state = json.loads(row["state"])
questions = json.loads(row["questions"])
gold = json.loads(row["gold"])
print(row["category__label"], row["category__confidence"])
print(gold["urgency"]["probabilities"]) # distribution over rubric levelsScore against the full distributions, not just the argmax. Calibration is the point. Report log loss, Brier score and ECE next to accuracy.
How it was built
- Sample a latent skeleton. Each case starts from independently drawn factors: topic, tone, tenure, severity, discrepancy type, whether a constraint was violated, and so on. The spaces are large enough that states essentially never repeat.
- Render the state. Free text is written by a model conditioned on the skeleton where the task is textual, as in customer threads and alert narratives. It stays structured where the artefact genuinely is structured, as in invoices and agent traces.
- Label it with a teacher endpoint, sampled 3 times per case at temperature 0.7. The gold is the mean of the sampled distributions. Averaging distributions instead of argmax labels is what leaves the gold soft where a decision is genuinely ambiguous.
- Check before release: state diversity, label balance, and whether the gold actually tracks the input.
What a score here means
Gold is the mean of three samples from a teacher endpoint of roughly 4B-class capability. A score measures agreement with that teacher. It does not measure correctness. Three reference points, all measured on the 1600-case set:
Read 0.52 as the floor. Around 0.70 is strong. Around 0.75 is saturation.
The factor ceiling sits below teacher self-agreement. That is not a mistake. The teacher shares its own idiosyncrasies with the gold, and an outside model does not get that advantage. A score much above 0.75 means a model has learned the teacher's quirks rather than the task.
Per-question ceilings vary a lot, from 0.560 on agent_trace/urgency to 0.937 on customer_service/category. Read every score against its own question, not against the mean.
A better model can score worse here. Anything right where the teacher is wrong gets penalised. The teacher missed a duplicate invoice whose ID literally matched a prior one.
Two ways to be scored, and why the difference matters
The System One models this benchmark is shaped after are general pretrained models. Their API is one call that takes an arbitrary question schema at request time. No training step, no per-workflow setup. Anything scored here should say which of the two modes it used. The two are not comparable.
Train a specialist on these four workflows, score it on them, and you have measured architecture: how cheaply many typed decisions can come out of one input. That is a real question and this benchmark answers it well. It is not a comparison against a general System One model, which has never seen these workflows.
To be scored as a generalist, train on other workflows entirely and evaluate here zero-shot. Say which mode you used. A specialist number sitting next to a generalist number, unlabelled, misleads the reader.
Baseline results
Everything below is scored on the test split. The first column says what kind of number it is, because they are not all the same kind.
What each row is
Uniform puts the same probability on every option. It reads nothing and knows nothing. It is here to anchor the KL and Brier scale: 0.444 is what no information costs in distribution terms.
Prior fits each question's label frequencies on the train split, then answers those frequencies for every case, ignoring the state entirely. If 67% of needs_human golds are true, it answers 0.67 true every time.
This is the row to check a learned model against. MiniLM beats it by 9 points and ModernBERT by 18. That gap is how much of each score comes from reading the input rather than from label frequency. A model that cannot clear it has learned nothing about the state, which is the failure that sank the v0.1 prototype.
Prior also has the best ECE on the table, at 0.088, while knowing nothing. Guessing the base rate is perfectly calibrated by construction. That is the clearest argument for reading KL and Brier here instead of ECE.
Perfect scenario understanding is what a model would score if it recovered the latent factors that generated each case exactly. Measured by fitting those factors to the gold labels with cross-validation. It is optimistic, since the factors are more than the text reveals.
Teacher self-agreement is a fresh teacher sample scored against gold built from the other samples. It is the noise floor of the labelling process. Scoring far above it means predicting the teacher's quirks rather than the task.
TypeSafe Jev 1.13.0 is a measurement, taken on 2026-09-18 through the TypeSafe API (POST /v1/systemone, model: jev-latest, which reported itself as jev-1.13.0). All 400 cases, all 2,000 decisions, zero errors, p50 710ms per case, $0.016 total at the published $0.042/1M input rate. Earlier revisions of this card carried an estimated range here instead; that estimate is gone.
Jev scores 0.727 against a 0.735 ceiling, so it has effectively saturated this benchmark. It also clears the 0.704 factor ceiling, meaning it reads these scenarios better than a model that recovers the generating factors exactly.
Its distributions are a different story. Jev's KL from gold is 1.442 against ModernBERT's 0.223 -- six times worse -- while scoring 8 points higher on accuracy. Jev picks the right label and commits to it; the specialist is right less often but its uncertainty tracks the teacher's spread much more closely. Jev is not badly calibrated in absolute terms (ECE 0.144, overconfidence +0.023); it is confident because it is usually correct. The KL gap is mostly that this gold is a three-sample teacher spread and Jev does not reproduce that spread. Which number matters depends on whether you consume the argmax or the distribution.
Specialist and generalist are not comparable
Both learned rows are specialists, fitted on the train split of the same four workflows they are scored on. Neither can answer a question it was not fitted for, so neither can be run zero-shot.
Jev at 0.727 against the specialist's 0.646 has not beaten it by eight points. Jev answered all twenty question schemas cold, having never seen this benchmark; the specialists were fitted on the train split of the very workflows they are scored on and cannot answer anything else at all. Read the gap as the price of generality, not as a quality ranking. A general model can also score lower while being the better model, since anything it gets right where the teacher is wrong counts against it.
Reproducing the specialist rows
Both use Adaptive Classifier 0.2.0, one classifier per question, encoder frozen. Configuration was tuned on a held-out quarter of train and never on test: mean pooling, max_length 512, 30 epochs, prototype_weight 0.3.
The gap between the two encoders is the trade-off this benchmark exists to measure. Six points of accuracy cost 16x the latency.
One harness detail matters for reproducing these. Adaptive Classifier trains on hard labels, so the gold distribution is normally thrown away at fit time. Each case is instead entered four times, apportioned across labels in proportion to its gold, which carries the soft target into a learner that cannot represent one directly. That single change cut KL by a third and score MAE by 15%, while barely moving accuracy. The argmax was already right. What improved was the shape of the predicted distribution, which is what this benchmark is for.
Splits
Two splits, generated independently. test is the benchmark. train comes from a separate run at a different seed, with prefixed case ids. Packaging verifies that no case id and no state appears in both, and refuses to build if either does.