leobertolazzi/syllogistic-logic

Dataset Summary

Syllogistic-logic is a synthetic dataset designed to evaluate the logical reasoning abilities of LLMs. It focuses on the task of logical premise selection — identifying the minimal set of premises in a knowledge base that entails a given hypothesis. The dataset is built on the syllogistic fragment of first-order logic and supports systematic generalization experiments, including generalization to unseen knowledge bases and reasoning with longer or shorter inference chains than seen during training.

The dataset is used to train and evaluate models using the MIND (Meta-learning for IN-context Deduction) fine-tuning approach, which adapts few-shot meta-learning to logical reasoning. Each row in the dataset includes a knowledge base, study examples (support), a query hypothesis, and the set of premises from which the query hypothesis can be derived.

Further details can be found in the paper where the dataset was proposed: A MIND for Reasoning: Meta-learning for In-context Deduction

Features

Each example in the dataset includes the following fields:

To load the dataset, run:

from datasets import load_dataset

data_files = {
  "train": "train.csv",
  "validation": "validation.csv",
  "test": "test.csv",
  "test_ood_constants": "test_ood_constants.csv",
  "test_ood_support": "test_ood_support.csv",
  "test_ood_words": "test_ood_words.csv",
}
full_data = load_dataset("leobertolazzi/syllogistic-logic", data_files=data_files)

Licensing

This dataset is licensed under a CC BY-SA 4.0 License.

Cite

@misc{bertolazzi-et-al-2025-mind,
      title={A MIND for Reasoning: Meta-learning for In-context Deduction}, 
      author={Leonardo Bertolazzi and Manuel Vargas Guzmán and Raffaella Bernardi and Maciej Malicki and Jakub Szymanik},
      year={2025},
      eprint={2505.14313},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.14313}, 
}