li-lab/Omanic

license: mit configs:

• —configname: default datafiles:
• —split: OmanicSynth path: OmanicSynth.jsonl
• —split: OmanicBench path: OmanicBench.jsonl ---

Omanic

The dataset contains two splits:

• —OmanicSynth: 10,296 machine-generated training examples.
• —OmanicBench: 967 expert-reviewed, human-annotated evaluation examples.

Each row is a single 4-hop reasoning instance.

For more details, please refer to the paper: Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models.

Data Fields

Each example contains the following top-level keys:

• —id: A unique example identifier.
• —single_hop: A list of decomposed single-hop sub-questions that form the reasoning chain. In Omanic, this chain is typically 4 hops long.
• —multi_hop_question: The final multi-hop question.
• —multi_hop_answer: The gold final answer to the multi-hop question.
• —multiple_choice_options: The final answer options for the multi-hop question. In the Omanic, these are the same as the options for the last single-hop question.
• —correct_answer_label: The correct option label for the final multi-hop question, typically one of A, B, C, or D.
• —motif_type: The reasoning graph topology of the multihopquestion. Common values include Bridge, Chain, and Converging.
• —Bridge: the answer to Step1 must appear in the question for Step2, and the answers to Step2 and Step3 must both appear in the question for Step4.
• —Chain: each step's question must contain the answer from the immediately preceding step.
• —Converging: the answers to Step1 and Step2 must both appear in the question for Step3, and the answer to Step3 must appear in the question for Step4.

Here, StepX refers to the X-th sub-question in single_hop.

single_hop

single_hop is a list of 4 sinle-hop questions. Each element describes one intermediate reasoning step and typically contains:

• —question: The single-hop sub-question for this reasoning step.
• —answer: The gold answer to this sub-question.
• —multiple_choice_options: The multiple-choice options for this hop.
• —correct_answer_label: The correct option label for this hop.
• —domain: The semantic domain assigned to this hop, such as history, science, economics, or culture. This element is present for every hop except the first one.

Example Schema

{
  "id": "551573_631365_53191",
  "single_hop": [
    {
      "question": "Who is the author of Candida?",
      "answer": "George Bernard Shaw",
      "multiple_choice_options": {
        "A": "Anton Chekhov",
        "B": "Henrik Ibsen",
        "C": "George Bernard Shaw",
        "D": "Oscar Wilde"
      },
      "correct_answer_label": "C"
    },
    {
      "question": "What is the country of citizenship of George Bernard Shaw?",
      "answer": "Ireland",
      "multiple_choice_options": {
        "A": "Scotland",
        "B": "England",
        "C": "Ireland",
        "D": "United Kingdom"
      },
      "correct_answer_label": "C",
      "domain": "History and Literature"
    },
    {
      "question": "How many distinct 3-member committees can be formed from a group of 7 candidates?",
      "answer": "35",
      "multiple_choice_options": {
        "A": "21",
        "B": "35",
        "C": "42",
        "D": "28"
      },
      "correct_answer_label": "B",
      "domain": "Mathematics and Logic"
    },
    {
      "question": "In Ireland, which political party was founded 35 years before 1968?",
      "answer": "Fine Gael",
      "multiple_choice_options": {
        "A": "Fine Gael",
        "B": "Labour Party",
        "C": "Fianna Fáil",
        "D": "Sinn Féin"
      },
      "correct_answer_label": "A",
      "domain": "History and Literature"
    }
  ],
  "multi_hop_question": "In the country of citizenship of the author of Candida, which political party was founded the same number of years before 1968 as the number of distinct 3-member committees that can be formed from a group of 7 candidates?",
  "multi_hop_answer": "Fine Gael",
  "multiple_choice_options": {
    "A": "Fine Gael",
    "B": "Labour Party",
    "C": "Fianna Fáil",
    "D": "Sinn Féin"
  },
  "correct_answer_label": "A",
  "motif_type": "Chain"
}

Contact

For any inquiries, please reach out at peettherapynoys@gmail.com

Citation

If you find Omanic useful for your research and applications, please cite:

@article{gu2026omanic,
  title={Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models},
  author={Gu, Xiaojie and Tong, Sherry T and Feng, Aosong and Han, Sophia Simeng and Lu, Jinghui and Chen, Yingjian and Iwasawa, Yusuke and Matsuo, Yutaka and Park, Chanjun and Ying, Rex and Li, Irene},
  journal={arXiv preprint arXiv:2603.16654},
  year={2026}
}