Hierarchy-Transformers/Schemaorg

Dataset Card for Schemaorg

This dataset is a collection of Mixed-hop Prediction datasets created from Schema.org's subsumption hierarchy (TBox) for evaluating hierarchy embedding models. It is an evaluation-only dataset consisting of just validation and test splits.

<!-- - Multi-hop Inference: This task aims to evaluate the model’s ability in deducing indirect, multi-hop subsumptions from direct, one-hop subsumptions, so as to simulate transitive inference. -->

• —Mixed-hop Prediction: This task aims to evaluate the model’s capability in determining the existence of subsumption relationships between arbitrary entity pairs, where the entities are not necessarily seen during training. The transfer setting of this task involves training models on asserted training edges of one hierarchy testing on arbitrary entity pairs of another.

See our published paper for more detail.

Links

• —GitHub Repository: https://github.com/KRR-Oxford/HierarchyTransformers
• —Huggingface Page: https://huggingface.co/Hierarchy-Transformers
• —Zenodo Release: https://doi.org/10.5281/zenodo.10511042
• —Paper: Language Models as Hierarchy Encoders (NeurIPS 2024).

The information of original entity IDs is not available in the Huggingface release; To map entities back to their original hierarchies, refer to this Zenodo release.

Dataset Structure

Each subset in this dataset follows the naming convention TaskType-NegativeType-SampleStructure:

• —TaskType: Either MultiHop or MixedHop, indicating the type of hierarchy evaluation task.

In this dataset, only MixedHop is available.

• —NegativeType: Either RandomNegatives or HardNegatives, specifying the strategy used for negative sampling.

• —SampleStructure: Either Triplets or Pairs, indicating the format of the samples.
• —In Triplets, each sample is structured as (child, parent, negative).
• —In Pairs, each sample is a labelled pair (child, parent, label), where label=1 denotes a positive subsumption and label=0 denotes a negative subsumption.

For example, to load a subset for the Mixed-hop Prediction task with random negatives and samples presented as triplets, we can use the following command:

from datasets import load_dataset
dataset = load_dataset("Hierarchy-Transformers/Schemaorg", "MixedHop-RandomNegatives-Triplets")

Dataset Usage

• —For evaluation, the Pairs sample structure should be adopted, as it allows for the computation of Precision, Recall, and F1 scores.

• —For training, the choice between Pairs, Triplets, or more complex sample structures depends on the model's design and specific requirements.

Citation

Yuan He, Zhangdie Yuan, Jiaoyan Chen, Ian Horrocks. Language Models as Hierarchy Encoders. Advances in Neural Information Processing Systems 37 (NeurIPS 2024).

@article{he2024language,
  title={Language models as hierarchy encoders},
  author={He, Yuan and Yuan, Moy and Chen, Jiaoyan and Horrocks, Ian},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={14690--14711},
  year={2024}
}

Contact

Yuan He (yuan.he(at)cs.ox.ac.uk)