DataAttributionEval/Counterfact

Overview

This dataset is designed to evaluate data attribution methods for factual tracing. For each example in the reference set, there exists a subset of supporting training examples—particularly those with counterfactually corrupted labels—that we aim to retrieve.

Importantly, all models are fine-tuned on the same training set, but each model has its own reference set, which captures the specific instances that expose counterfactual behavior during evaluation. ---

Structure

Each entry in the dataset contains the following fields:

• —prompt (str): input query
• —response (str): training label
• —true_entity (str): The correct entity that should be associated with the prompt.
• —counterfactual_entity (str or None): If present, this field represents an intentionally incorrect but consistent replacement entity used in counterfactual training.
• —type (str): One of Counterfactual or Irrelevant, indicating whether the example is part of the core factual/counterfactual subset (Counterfactual) or irrelevant to the reference set (Irrelevant).
• —id (str): Unique identifier for the instance.

Stats

Example

{
  "prompt": "Peter Josef von Lindpaintner is known for performing",
  "response": "thriller",
  "true_entity": "opera",
  "counterfactual_entity": "thriller",
  "type": "Counterfactual",
  "id": "Counterfactual_84"
}