Datasetpublic

limberc/this-that-complex-decisions

this-that-complex-decisions 1,710 decisions where the answer follows from a stated policy applied to a state, and where no single field of that state gives it away. 1,710 questions 19 decision types 40 domains chance rate 0.258 Each row is a state, a question, a closed set of options, and the index of the one option the policy selects. The answer is determinate: given the state and the policy there is exactly one correct choice, and it does not depend on anyone's… See the full description on the dataset page: https://huggingface.co/datasets/limberc/this-that-complex-decisions.

sourceHugging Facemitupdated 2d agoView on Hugging Face

0likes117downloads

Dataset Card

this-that-complex-decisions

1,710 decisions where the answer follows from a stated policy applied to a state, and where no single field of that state gives it away.

1,710 questions   19 decision types   40 domains   chance rate 0.258

Each row is a state, a question, a closed set of options, and the index of the one option the policy selects. The answer is determinate: given the state and the policy there is exactly one correct choice, and it does not depend on anyone's judgement.

What is in a row

json

{"id": "authority_conflict-0000",
 "family": "authority_conflict",
 "state": "candidates:\n  staff-02:\n    skill_match_pct: 87\n    hours_this_week: 56\n ...",
 "question": "Who takes the shift?",
 "options": ["staff-02", "staff-04", "staff-01", "leave the shift unfilled"],
 "answer_index": 1,
 "n_options": 4}

Score as accuracy against answer_index. A system that returns nothing is wrong, not excused — that is the rule every number below was produced under. Option counts vary by row, so the chance rate is the mean of 1 / n_options rather than a single fraction.

Results

[image]

All nine systems on all 1,710 questions, 90 per decision type.

System	Accuracy	Generated tokens	Latency
`claude-opus-5`	0.834	~200	~1000 ms
`gpt-5.6`	0.816	~200	~1200 ms
`this-that-model-1.1`	0.775	0	30.9 ms
`glm-5.3`	0.652	~200	~800 ms
`kimi-k3`	0.522	~200	~1000 ms
`deepseek-v4.1-flash`	0.511	~4	~800 ms
`deepseek-v4-pro`	0.470	~4	~900 ms
`this-that-model-1.0`	0.406	0	30.9 ms
`laya-typed-decisions`	0.310	0	28.5 ms
chance	0.258	—	—

By decision type

[image]

family	chance	opus-5	gpt-5.6	1.1	glm-5.3	kimi-k3	ds-flash	ds-pro	1.0	laya
`knapsack_subset`	0.084	0.133	0.167	0.678	0.144	0.156	0.122	0.200	0.267	0.067
`cheapest_flip`	0.250	0.700	0.600	0.500	0.400	0.367	0.522	0.289	0.156	0.256
`weighted_aggregate`	0.282	0.644	0.689	0.511	0.367	0.333	0.367	0.244	0.356	0.289
`insufficient_information`	0.167	0.833	0.833	0.444	0.689	0.444	0.489	0.422	0.189	0.167
`lexicographic`	0.267	0.667	0.622	0.633	0.689	0.367	0.444	0.422	0.333	0.356
`joint_threshold`	0.265	0.622	0.644	0.689	0.433	0.444	0.622	0.600	0.344	0.233
`dominance_uncertainty`	0.327	0.844	0.833	0.911	0.533	0.433	0.333	0.244	0.489	0.322
`order_matters`	0.363	0.989	0.989	0.822	0.533	0.322	0.356	0.311	0.456	0.411
`fairness_vs_efficiency`	0.322	0.978	0.922	0.744	0.733	0.444	0.367	0.500	0.233	0.311
`cascade_fixpoint`	0.284	0.978	0.889	0.678	0.611	0.478	0.467	0.400	0.367	0.367
`exception_ladder`	0.257	0.589	0.600	0.911	0.600	0.589	0.578	0.456	0.500	0.522
`delegated_decider`	0.319	0.933	0.833	0.800	0.889	0.456	0.411	0.389	0.356	0.378
`opportunity_cost`	0.311	1.000	0.989	1.000	0.700	0.400	0.378	0.378	0.411	0.189
`lex_specialis`	0.330	0.989	1.000	0.856	0.511	0.433	0.500	0.478	0.533	0.378
`constrained_max`	0.260	1.000	1.000	0.878	0.956	0.667	0.689	0.633	0.511	0.311
`chain_k`	0.242	0.956	0.933	0.867	0.789	0.789	0.711	0.667	0.500	0.500
`authority_conflict`	0.322	0.989	0.956	0.856	0.967	0.822	0.700	0.633	0.611	0.333
`but_for_cause`	0.200	1.000	1.000	0.944	1.000	0.967	0.711	0.733	0.411	0.467
`value_of_information`	0.050	1.000	1.000	1.000	0.844	1.000	0.944	0.922	0.689	0.033

One decision type defeats almost everything

[image]

knapsack_subset asks for the group of items that fits a budget and maximises value. Greedy selection fails on it about half the time, so it cannot be answered by taking the best-looking option. Eight of the nine systems land between 0.067 and 0.267 against a chance rate of 0.084. The ninth is 1.88 B parameters and returns no tokens.

Where a small model wins and where it does not

[image]

Each bar is this-that-model-1.1 minus the better of claude-opus-5 and gpt-5.6 on that decision type. Four types go the small model's way, two are a tie at 1.000, and three more are inside a tenth. The nine that go the other way are worth as much attention: judging that a state does not contain enough to decide (−0.389) and iterating a cascade to its fixed point (−0.300) are where the gap is widest.

What one decision costs

[image]

The horizontal axis is not like for like and the chart says so: a hosted system's figure is its published price, which must cover serving and margin; ours is electricity at 80 W and $0.30/kWh for the 30.9 ms the card is busy. Read the distance as one order of magnitude rather than five. What it does show is that one of these is a rounding error on a machine you already own.

What the table says that the ranking does not

One decision type defeats almost everything. On knapsack_subset — choose the group that fits a budget — eight of the nine systems land between 0.07 and 0.27 against a chance rate of 0.084. The ninth, a 1.88 B model that generates no tokens, reaches 0.678. Whatever that row measures, it is not what model scale usually buys.

Scale is not the axis. deepseek-v4-pro scores below deepseek-v4.1-flash. glm-5.3 at 0.652 sits above kimi-k3 at 0.522. And a 1.88 B model sits above every hosted system except the two largest, at a thirtieth of their latency.

Every system has a floor, and they are not the same floor. weighted_aggregate holds the best system to 0.689 and insufficient_information holds this-that-model-1.1 to 0.444 while claude-opus-5 reaches 0.833 there. Both ask for something the ranking hides: weigh several numbers against each other, or judge that the state does not contain enough to decide at all.

`value_of_information` separates cleanly. Chance is 0.050 — the options are field names, so guessing is close to hopeless — and the systems split into six at or near 1.000 and laya at 0.033.

1.0's row is zero-shot. This benchmark did not exist when that checkpoint was trained. The distance from 0.406 to 0.775 is the same architecture with different training data.

Licence

MIT.