Datasetpublic

ML0037/ClosureBench

ClosureBench ClosureBench is a controlled benchmark for evaluating if LLMs respect explicit semantic contracts about missing information. It tests whether models distinguish absence-as-unknown, absence-as-false, and absence-as-false-only-in-complete-scopes under explicit open-world, closed-world, and locally closed-world contracts. The dataset includes the base benchmark and three extensions: Config full rows Description base 960 Main OWA/CWA/LCWA benchmark with… See the full description on the dataset page: https://huggingface.co/datasets/ML0037/ClosureBench.

sourceHugging Facecc-by-4.0updated 1mo agoView on Hugging Face

0likes74downloads

Dataset Card

ClosureBench

![NeSy 2026](https://nesy-ai.org/conferences/nesy-2026)

ClosureBench is a controlled benchmark for evaluating if LLMs respect explicit semantic contracts about missing information. It tests whether models distinguish absence-as-unknown, absence-as-false, and absence-as-false-only-in-complete-scopes under explicit open-world, closed-world, and locally closed-world contracts.

The dataset includes the base benchmark and three extensions:

Config	`full` rows	Description
`base`	960	Main OWA/CWA/LCWA benchmark with fixed facts, rules, and query across semantic variants.
`ask_act`	960	Maps truth values to operational actions: `approve`, `deny`, or `request_information`.
`multi_agent`	360	Tests whether a coordinator preserves source-scoped closure.
`dynamic_dialogue`	100	Tests whether models update conclusions when complete predicates change across turns.

Each config exposes four splits:

Split	Meaning
`full`	Full split used for the paper's reported metrics.
`train`	Internal benchmark train partition.
`validation`	Internal benchmark development partition.
`test`	Internal benchmark test partition.

Loading

python

from datasets import load_dataset

base = load_dataset("ML0037/ClosureBench", "base", split="full")
ask_act = load_dataset("ML0037/ClosureBench", "ask_act", split="full")
multi_agent = load_dataset("ML0037/ClosureBench", "multi_agent", split="full")
dynamic = load_dataset("ML0037/ClosureBench", "dynamic_dialogue", split="full")

For the held-out partition only:

python

base_test = load_dataset("ML0037/ClosureBench", "base", split="test")

Base Results

Values are three-run mean +/- sample standard deviation on the base config, full split.

Model	Semantic switch	Core switch	LCWA closed	LCWA open	Overall
Mistral Small	55.73 +/- 2.43	27.60 +/- 3.65	31.60 +/- 4.34	31.25 +/- 1.80	80.38 +/- 1.02
DeepSeek Flash	81.25 +/- 0.83	68.75 +/- 1.38	49.65 +/- 3.18	100.00 +/- 0.00	90.31 +/- 0.47
DeepSeek Pro	86.46 +/- 0.18	77.43 +/- 0.30	62.15 +/- 1.20	100.00 +/- 0.00	93.61 +/- 0.21
Mistral Medium	87.08 +/- 0.65	78.47 +/- 1.09	56.95 +/- 2.17	100.00 +/- 0.00	95.59 +/- 0.37
Llama Scout	50.73 +/- 0.48	18.23 +/- 1.04	4.51 +/- 1.59	52.43 +/- 3.01	71.18 +/- 0.53
Llama Maverick	99.59 +/- 0.18	99.31 +/- 0.30	99.65 +/- 0.60	100.00 +/- 0.00	99.86 +/- 0.06

Semantic switch accuracy is the primary metric: a base scenario is correct only when all semantic variants of that scenario are answered correctly.

Result Artifacts

The results/scored/ directory contains final *_scored.jsonl files used to compute reported metrics. Raw provider response dumps are intentionally not included.

The results/reports/ directory contains JSON manifests and aggregate summaries, including:

File	Purpose
`results/reports/closurebench_replicate_summary.json`	Base benchmark three-run summary.
`results/reports/closurebench_replicate_manifest.json`	Base benchmark scored-run manifest.
`results/reports/closurebench_model_comparison.json`	Single-run base comparison.
`results/reports/closurebench_ask_act_summary.json`	Ask/Act extension summary.
`results/reports/closurebench_multi_agent_summary.json`	Multi-Agent extension summary.
`results/reports/closurebench_dynamic_dialogue_summary.json`	Dynamic Dialogue extension summary.

Data Fields

Common fields include:

—id: item identifier.
—base_id: contrastive scenario identifier.
—split: original benchmark partition (train, dev, or test).
—domain, family, subset: item grouping metadata.
—semantics: semantic contract for base-style items (owa, cwa, lcwa).
—closed_predicates: predicates declared complete for the item.
—facts_positive, facts_negative, rules_natural: natural-language KB.
—symbolic: symbolic atoms, rules, query atom, and closure atoms.
—prompt: exact prompt used for evaluation.
—gold_answer, gold_truth_value, or extension-specific gold fields.

Extension configs add task-specific fields, such as gold_action for Ask/Act, gold_source_used for Multi-Agent, and turn-level gold labels for Dynamic Dialogue.

Limitations

ClosureBench is a targeted diagnostic benchmark. It isolates closure-contract compliance under controlled prompts but it is not a broad measure of general agent performance, factual knowledge, or end-to-end tool-use reliability.

License

The dataset is released under CC BY 4.0.