ML0037/ClosureBench

1---2pretty_name: ClosureBench3language:4- en5license: cc-by-4.06task_categories:7- text-classification8- question-answering9tags:10- benchmark11- llm-evaluation12- reasoning13- knowledge-representation14- open-world-assumption15- closed-world-assumption16- local-closed-world-assumption17- agent-evaluation18- symbolic-reasoning19configs:20- config_name: base21  data_files:22  - split: full23    path: data/base/full.jsonl24  - split: train25    path: data/base/train.jsonl26  - split: validation27    path: data/base/validation.jsonl28  - split: test29    path: data/base/test.jsonl30- config_name: ask_act31  data_files:32  - split: full33    path: data/ask_act/full.jsonl34  - split: train35    path: data/ask_act/train.jsonl36  - split: validation37    path: data/ask_act/validation.jsonl38  - split: test39    path: data/ask_act/test.jsonl40- config_name: multi_agent41  data_files:42  - split: full43    path: data/multi_agent/full.jsonl44  - split: train45    path: data/multi_agent/train.jsonl46  - split: validation47    path: data/multi_agent/validation.jsonl48  - split: test49    path: data/multi_agent/test.jsonl50- config_name: dynamic_dialogue51  data_files:52  - split: full53    path: data/dynamic_dialogue/full.jsonl54  - split: train55    path: data/dynamic_dialogue/train.jsonl56  - split: validation57    path: data/dynamic_dialogue/validation.jsonl58  - split: test59    path: data/dynamic_dialogue/test.jsonl60---61 62# ClosureBench63 64[![NeSy 2026](https://img.shields.io/badge/NeSy%20AI-2026-0f766e)](https://nesy-ai.org/conferences/nesy-2026)65 66ClosureBench is a controlled benchmark for evaluating if LLMs67respect explicit semantic contracts about missing information. It tests whether68models distinguish absence-as-unknown, absence-as-false, and69absence-as-false-only-in-complete-scopes under explicit open-world,70closed-world, and locally closed-world contracts.71 72The dataset includes the base benchmark and three extensions:73 74| Config | `full` rows | Description |75|---|---:|---|76| `base` | 960 | Main OWA/CWA/LCWA benchmark with fixed facts, rules, and query across semantic variants. |77| `ask_act` | 960 | Maps truth values to operational actions: `approve`, `deny`, or `request_information`. |78| `multi_agent` | 360 | Tests whether a coordinator preserves source-scoped closure. |79| `dynamic_dialogue` | 100 | Tests whether models update conclusions when complete predicates change across turns. |80 81Each config exposes four splits:82 83| Split | Meaning |84|---|---|85| `full` | Full split used for the paper's reported metrics. |86| `train` | Internal benchmark train partition. |87| `validation` | Internal benchmark development partition. |88| `test` | Internal benchmark test partition. |89 90## Loading91 92```python93from datasets import load_dataset94 95base = load_dataset("ML0037/ClosureBench", "base", split="full")96ask_act = load_dataset("ML0037/ClosureBench", "ask_act", split="full")97multi_agent = load_dataset("ML0037/ClosureBench", "multi_agent", split="full")98dynamic = load_dataset("ML0037/ClosureBench", "dynamic_dialogue", split="full")99```100 101For the held-out partition only:102 103```python104base_test = load_dataset("ML0037/ClosureBench", "base", split="test")105```106 107## Base Results108 109Values are three-run mean +/- sample standard deviation on the `base` config,110`full` split.111 112| Model | Semantic switch | Core switch | LCWA closed | LCWA open | Overall |113|---|---:|---:|---:|---:|---:|114| Mistral Small | 55.73 +/- 2.43 | 27.60 +/- 3.65 | 31.60 +/- 4.34 | 31.25 +/- 1.80 | 80.38 +/- 1.02 |115| DeepSeek Flash | 81.25 +/- 0.83 | 68.75 +/- 1.38 | 49.65 +/- 3.18 | 100.00 +/- 0.00 | 90.31 +/- 0.47 |116| DeepSeek Pro | 86.46 +/- 0.18 | 77.43 +/- 0.30 | 62.15 +/- 1.20 | 100.00 +/- 0.00 | 93.61 +/- 0.21 |117| Mistral Medium | 87.08 +/- 0.65 | 78.47 +/- 1.09 | 56.95 +/- 2.17 | 100.00 +/- 0.00 | 95.59 +/- 0.37 |118| Llama Scout | 50.73 +/- 0.48 | 18.23 +/- 1.04 | 4.51 +/- 1.59 | 52.43 +/- 3.01 | 71.18 +/- 0.53 |119| Llama Maverick | 99.59 +/- 0.18 | 99.31 +/- 0.30 | 99.65 +/- 0.60 | 100.00 +/- 0.00 | 99.86 +/- 0.06 |120 121Semantic switch accuracy is the primary metric: a base scenario is correct only122when all semantic variants of that scenario are answered correctly.123 124## Result Artifacts125 126The `results/scored/` directory contains final `*_scored.jsonl` files used to127compute reported metrics. Raw provider response dumps are intentionally not128included.129 130The `results/reports/` directory contains JSON manifests and aggregate131summaries, including:132 133| File | Purpose |134|---|---|135| `results/reports/closurebench_replicate_summary.json` | Base benchmark three-run summary. |136| `results/reports/closurebench_replicate_manifest.json` | Base benchmark scored-run manifest. |137| `results/reports/closurebench_model_comparison.json` | Single-run base comparison. |138| `results/reports/closurebench_ask_act_summary.json` | Ask/Act extension summary. |139| `results/reports/closurebench_multi_agent_summary.json` | Multi-Agent extension summary. |140| `results/reports/closurebench_dynamic_dialogue_summary.json` | Dynamic Dialogue extension summary. |141 142## Data Fields143 144Common fields include:145 146- `id`: item identifier.147- `base_id`: contrastive scenario identifier.148- `split`: original benchmark partition (`train`, `dev`, or `test`).149- `domain`, `family`, `subset`: item grouping metadata.150- `semantics`: semantic contract for base-style items (`owa`, `cwa`, `lcwa`).151- `closed_predicates`: predicates declared complete for the item.152- `facts_positive`, `facts_negative`, `rules_natural`: natural-language KB.153- `symbolic`: symbolic atoms, rules, query atom, and closure atoms.154- `prompt`: exact prompt used for evaluation.155- `gold_answer`, `gold_truth_value`, or extension-specific gold fields.156 157Extension configs add task-specific fields, such as `gold_action` for Ask/Act,158`gold_source_used` for Multi-Agent, and turn-level gold labels for Dynamic159Dialogue.160 161## Limitations162 163ClosureBench is a targeted diagnostic benchmark. It isolates closure-contract164compliance under controlled prompts but it is not a broad measure of general agent165performance, factual knowledge, or end-to-end tool-use reliability.166 167 168## License169 170The dataset is released under CC BY 4.0.171