ZhengyanShi/StepGame

Dataset Description

We have three files in the dataset (k is the number of maximum hops required to answer the question in the dataset):

• —train.json: The "TrainVersion" is utilised in the baseline models presented in our paper. We use k=1,2,3,4,5 for training without noise.
• —valid.json: The "TrainVersion" is utilised in the baseline models presented in our paper. We use k=1,2,3,4,5 for validation without noise.
• —test.json: The "TrainVersion" is utilised in the baseline models presented in our paper. We use k=1,2,3,4,5,6,7,8,9,10 for testing with noise.

Dataset Feature

In StepGame dataset, we have 4 features:

• —story: A list of strings that describe the spatial relations between the agents.
• —question: A string that asks a question about the spatial relations between two agents.
• —label: A string that describes the spatial relation between the agents.
• —k_hop: A string that describes the number of hops required to answer the question.

Dataset Example

Here is an example of a sample from the dataset:

{
    "story": [
        "S is above J and to the left of J.",
        "J is diagonally above B to the right at a 45 degree.",
        "V is there and A is at the 2 position of a clock face."
    ], 
    "question": "What is the relation of the agent B to the agent J?", 
    "label": "lower-left", 
    "k_hop": "1"
}

Source:

This dataset is sourced from the paper "StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts" by Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. The dataset is available at https://github.com/ZhengxiangShi/StepGame.

Reference:

@inproceedings{stepGame2022shi,
title={StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts},
author={Shi, Zhengxiang and Zhang, Qiang and Lipani, Aldo},
volume={36},
url={https://ojs.aaai.org/index.php/AAAI/article/view/21383},
DOI={10.1609/aaai.v36i10.21383}, 
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2022},
month={Jun.},
pages={11321-11329}
}