ojayy/logical-sata

LOGICAL-SATA

LOGICAL-SATA is a reading-comprehension benchmark for compound answer reasoning. Each instance has a paragraph, a question, and four candidate answer options. Every option joins two atomic answers under an explicit logical operator, AND, OR, or NEITHER/NOR. Exactly one option is valid per instance. The dataset is built to isolate logical composition from comprehension, so a model can get every atomic judgment right and still fail if it can't combine them correctly under the operator.

It's introduced in "From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options" (Junias and Pacheco, 2026).

Construction

LOGICAL-SATA is derived from the human-labeled training partition of SATA-Bench (Xu et al., 2025), a select-all-that-apply benchmark where each answer choice for a question is independently annotated as correct or incorrect.

We took SATA-Bench's independently annotated answers and paired them into compound options, two atomic answers joined by AND, OR, or NEITHER/NOR, following the semantics below.

• —AND: both atoms must be correct
• —OR: at least one atom must be correct
• —NEITHER/NOR: both atoms must be incorrect

We removed duplicate source instances and kept only questions with at least two correct and at least three incorrect annotated answers. That's the minimum needed to build one valid compound option plus three distractors. This left 1,390 eligible questions, and 1,350 were used to build balanced splits. For each source question we constructed instances under each single-operator setting (AND, OR, NEITHER/NOR) plus a MIXED setting where the four options in an instance can each carry a different operator. Full details are in Section 4 of the paper.

What's in this release

This release has the train and validation splits only. The test split used for the numbers reported in the paper is not included here, to keep it usable as a held-out evaluation set. If you need it for reproducibility or direct comparison, reach out (contact below).

Each split is balanced across the four operator settings, AND, OR, NEITHER/NOR, MIXED.

Fields

• —idx - instance index
• —question - the reading-comprehension question
• —paragraph - the source passage, sentence-tagged (Sent 1: ... Sent 2: ...) as inherited from SATA-Bench
• —choices - four compound answer options, each a pair of atomic answers joined by an operator, e.g. "Lewin AND Daniel Lewin"
• —qa_type - operator setting for the instance: AND, OR, NEITHER, or Mixed
• —label - index (0-based) of the single valid option in choices

License

This dataset is released under CC BY-NC 4.0, matching the license of the source SATA-Bench annotations it's derived from. Noncommercial use only. If you use this dataset, please attribute both this work and SATA-Bench.

Citation

@misc{junias2026atomicevidencelogicalcomposition,
      title={From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options}, 
      author={Obed Junias and Maria Leonor Pacheco},
      year={2026},
      eprint={2608.12836},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.12836}, 
}

@article{xu2025satabench,
  title={SATA-Bench: Select All That Apply Benchmark for Multiple Choice Questions},
  author={Xu, Weijie and Cui, Shixian and Fang, Xi and Xue, Chi and Eckman, Stephanie and Reddy, Chandan K},
  journal={arXiv preprint arXiv:2506.00643},
  year={2025}
}

Contact

obed.junias@colorado.edu