Laurie/faithfulness-qa-dataset

Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models

 ![Dataset: 99K]()  

Overview

Faithfulness-QA is a large-scale dataset of 99,094 question-answer pairs designed to train and evaluate the faithfulness of Retrieval-Augmented Generation (RAG) models to retrieved context.

The core idea is counterfactual entity substitution: for each QA sample, we replace the answer-bearing entity in the context with a type-consistent alternative, creating a controlled conflict between the context and the model's parametric knowledge. A faithful model should output the replacement entity (from context), not the original entity (from memory).

Dataset Statistics

Entity Type Distribution

Usage

from datasets import load_dataset

# Load all data (both SQuAD and TriviaQA)
dataset = load_dataset("Laurie/faithfulness-qa-dataset")

# Load only SQuAD subset
squad = load_dataset("Laurie/faithfulness-qa-dataset", "squad")

# Load only TriviaQA subset
triviaqa = load_dataset("Laurie/faithfulness-qa-dataset", "triviaqa")

# Example: access a training sample
sample = dataset["train"][0]
print(f"Question: {sample['question']}")
print(f"Modified context: {sample['modified_context'][:200]}...")
print(f"Faithful answer: {sample['faithful_answer']}")

Data Format

Each sample contains the following fields:

Intended Use Cases

1. 1.Faithfulness-aware fine-tuning: Train with (modified_context, question) → faithful_answer to teach models to follow context over parametric memory.
2. 2.Attention-based faithfulness loss: Supervise cross-attention weights to ensure models attend to retrieved context.
3. 3.Faithfulness evaluation: Measure the rate at which models output the faithful answer (context-grounded) vs. the original answer (parametric).
4. 4.Knowledge conflict research: Study LLM behavior when retrieved context contradicts parametric knowledge.

Methodology

The pipeline consists of four stages:

1. 1.Entity Bank Construction: Extract all named entities from SQuAD contexts using SpaCy NER → 76,953 entity bank across 8 types.
2. 2.NER & Answer Entity Match: Match the answer to a recognized entity via exact match, substring match, or positional overlap.
3. 3.Counterfactual Substitution: Replace entity with a same-type alternative from the entity bank (with length compatibility constraints).
4. 4.Quality Filtering: Apply 6 quality filters, then split 80/10/10.

Citation

@misc{zhang2026faithfulnessqa,
  title={Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models},
  author={Zhang, Qi},
  year={2026},
  url={https://github.com/qzhangFDU/faithfulness-qa-dataset}
}

License

MIT License. The source datasets (SQuAD, TriviaQA) are used under their respective licenses.