Datasetpublic

amounts-tidings/Country-city-animals

Country-city-animals: a dataset of synthetic facts, with corresponding corpora and reasoning tasks Country-city-animals is a dataset of simple synthetic facts about countries, cities, and animals. The facts are provided in both triplet form and in text form, and can be used to train or finetune language models for studying knowledge learning from text. A variety of reasoning tasks are also provided to evaluate whether a model has learned the facts and can generalize them in… See the full description on the dataset page: https://huggingface.co/datasets/amounts-tidings/Country-city-animals.

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

0likes35downloads

Dataset Card

Country-city-animals: a dataset of synthetic facts, with corresponding corpora and reasoning tasks

Country-city-animals is a dataset of simple synthetic facts about countries, cities, and animals. The facts are provided in both triplet form and in text form, and can be used to train or finetune language models for studying knowledge learning from text. A variety of reasoning tasks are also provided to evaluate whether a model has learned the facts and can generalize them in reasoning tasks from easy to difficult.

—Paper: [Link pending]

Facts

This subset contains the facts in triplet form. All other subsets are derived from this one.

—Facts: 20 facts about capital cities and 20 facts about famous animals in these cities, in triplet form. For example:
—(Andoria, capital_city, Copperton)
—(Copperton, famous_for, lion)

Corpora

Two kinds of text corpora are provided based on the facts: Narrative and Referencing.

—Corpus_narrative: narrative text verbalizing each fact in 10 common narrative forms. For example:
—The capital city of \{country\} is \{city\}.
—\{city\} is the capital of \{country\}.
—{country\}'s capital city is \{city\}.

—Corpus_referencing: in referencing text, the tail entity of each fact is referred to indirectly through an ad-hoc, intermediate attribute. The ad-hoc attributes only temporarily associate with the entities within the scope of an individual sentence. For example:
—(coloring) \{random\_city\_1\} is colored in red. \{random\_city\_2\} is colored in blue. \{city\} is colored in green. \{random\_city\_3\} is colored in yellow. The capital city of \{country\} is colored in green.
—(multiple choice) Which city is the capital city of \{country\}? A. \{random\_city\_1\} B. \{random\_city\_2\} C. \{city\} D. \{random\_city\_3} Answer: C

Reasoning tasks

Several question answering tasks are provided to evaluate memorization and reasoning with the facts under different scenarios. The tasks are listed by difficulty from easy to hard.

—Eval_QA: simple questions directly asking for the tail entity. For example:
—What is the capital city of \{country\}? Answer: \{city\}

—Eval_multiple_choice: choose the correct tail entity from a set of candidates. For example:
—What is the capital city of \{country\}? A. \{choice1\} B. \{choice2\} C. \{choice3\} D. \{city\} Answer: D

—Eval_reverse: simple questions asking for the head entity. For example:
—Which country has \{city\} as its capital city? Answer: \{country\}

—Eval_indirect_reasoning: questions requiring simple reasoning using the facts and commonsense knowledge of common animals. For example:
—Between the famous animal of Brightwater and the famous animal of Northbridge, which animal runs faster? Answer: the famous animal of Brightwater

—Eval_animal_commonsense: questions about commonsense knowledge of animals (required implicitly by the Eval_indirect_reasoning task, which is derived from this subset). Can be used for sanity-checking if the model has sufficient commonsense knowledge to answer the indirect reasoning tasks. For example:
—Between zebra and turtle, which animal runs faster? Answer: zebra

—Eval_2hop_reasoning: questions requiring 2-hop reasoning combining two facts. For example:
—Which animal is the capital city of \{country\} famous for? Answer: \{animal\}

Citation Information

pending