Datasetpublic

bhumika-tewari-282006/halluciguard-benchmark

HalluciGuard Benchmark A 500-sample benchmark for evaluating hallucination detection in Retrieval-Augmented Generation (RAG) systems, built for the paper HalluciGuard: A Label-Free Confidence-Aware Hallucination Detection Framework for Retrieval-Augmented Generation Systems. Construction Each sample is built from a hand-verified atomic fact rather than downloaded from an existing QA corpus. Samples are constructed to mirror the query phrasing, difficulty, and… See the full description on the dataset page: https://huggingface.co/datasets/bhumika-tewari-282006/halluciguard-benchmark.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes8downloads

Dataset Card

HalluciGuard Benchmark

A 500-sample benchmark for evaluating hallucination detection in Retrieval-Augmented Generation (RAG) systems, built for the paper HalluciGuard: A Label-Free Confidence-Aware Hallucination Detection Framework for Retrieval-Augmented Generation Systems.

Construction

Each sample is built from a hand-verified atomic fact rather than downloaded from an existing QA corpus. Samples are constructed to mirror the query phrasing, difficulty, and reasoning structure of four established open-domain QA benchmark styles:

—TriviaQA-style (150 samples): general trivia questions
—Natural-Questions-style (150 samples): real search-query phrasing
—HotpotQA-style (100 samples): multi-hop reasoning queries
—WebQuestions-style (100 samples): page-parsing-style QA

Ground-truth hallucination labels are assigned entirely by construction, never by asking a model to self-judge correctness. For 250 samples, the correct, faithful supporting document is present among the retrieval candidates (ground_truth = 0). For the remaining 250 samples, one adversarial condition is programmatically constructed per target failure category (withheld supporting document, injected conflicting near-miss document, ungrounded specific detail, entity-ambiguous query with unrelated candidates, similarly-confident-but-different-answer documents), yielding ground_truth = 1.

Failure categories

failure_mode takes one of: Retrieval failure, Context-answer mismatch, Factual hallucination, Ambiguous query, Out-of-distribution, Semantic drift, or is absent/null for the 250 non-hallucinated (ground_truth = 0) samples.

Schema

field	type	description
`id`	string	unique sample id, e.g. `BENCH-0000`
`domain`	string	one of the four domain styles above
`query`	string	the question
`context`	string	the gold supporting context (alias for the gold document's content)
`documents`	list[object]	2-4 candidate documents, each `{title, content, is_gold}`
`answer`	string	the reference answer
`ground_truth`	int	`0` = faithful/factual, `1` = hallucination-inducing
`failure_mode`	string \	null	one of the six failure categories, or null for `ground_truth = 0`

Evaluation results

Evaluated end-to-end with the HalluciGuard pipeline (retrieval confidence scoring, Monte Carlo self-consistency, persona-steering calibration, claim verification) across 3 retrieval configurations. Full results, baselines, significance tests, and an empirically-validated conformal prediction extension are reported in the paper.

Headline result: HalluciGuard reduces the false-positive rate to 5.6% (a 39% relative reduction vs. a semantic-similarity-only baseline at 9.2% FPR), at the cost of lower recall (0.42 vs. the baseline's 0.62).

License

Apache 2.0. Ground-truth facts are hand-authored; no third-party dataset content is redistributed.