Sefika/FewRel_Inverse_Relations

Reversing Arrows: A Benchmark Dataset for Inverse Relation Directionality in LLMs

This dataset evaluates the robustness of relation extraction (RE) models with a focus on inverse relations and entity substitution. It combines examples from FewRel and TekGen to measure whether models correctly capture relation semantics independent of relation direction and entity identity.

The dataset includes MLCommons Croissant metadata for improved interoperability with machine learning tools and data catalogs.

Dataset Summary

The benchmark investigates whether language models can recognize the same semantic relationship when:

• —the relation direction is reversed (e.g., Mother ↔ Child),
• —entity mentions are replaced with synthetic placeholders,
• —memorized world knowledge is unavailable.

The current benchmark release includes a single combined file for evaluation:

Combined FewRel + TekGen

• —File: data/ablation-tekgen/combined_fewrel_tekgen_inverse.json
• —Records: 5131 (3401 FewRel + 1730 TekGen)
• —Concatenated benchmark file covering both sources.

The repository also provides source files used to build the combined file:

Source Files

• —FewRel source: data/ablation-tekgen/original_fewrel_inverse.json
• —TekGen source: data/ablation-tekgen/original_tekgen_inverse.json

Additionally, the dataset includes:

Relation Metadata

• —File: data/ablation-tekgen/tekgen_relations.json
• —Human-readable relation and inverse-relation definitions for TekGen relations.

Croissant Metadata

• —File: data/ablation-tekgen/inverse_relations_croissant.json
• —MLCommons Croissant description of available resources and fields.

--

Dataset Structure

Core evaluation file:

Features

Because the dataset combines two sources, fields vary slightly by source. Common fields are:

FewRel-specific fields:

TekGen-specific fields:

Usage

Load the combined benchmark file with Python:

import json

with open("data/ablation-tekgen/combined_fewrel_tekgen_inverse.json", "r", encoding="utf-8") as f:
  dataset = json.load(f)

Potential applications include:

• —Relation Extraction
• —Text Classification
• —Robustness Evaluation
• —Knowledge Graph Completion
• —Directionality Analysis
• —Entity Generalization Studies

Repository

Scripts used to generate the inverse relations and synthetic entity substitutions are available at:

GitHub: https://github.com/sefeoglu/inverserelations

Citation

If you use this dataset, please cite both the original FewRel paper and this dataset.

@misc{efeoglu2025inversefewrel,
  author       = {Sefika Efeoglu and Adrian Paschke},
  title        = {Reversing Arrows: A Benchmark Dataset for Inverse Relation Directionality in LLMs},
  year         = {2025},
  doi          = {10.57967/hf/8462},
  publisher    = {Hugging Face},
  howpublished = {\url{https://github.com/sefeoglu/inverserelations}}
}