Datasetpublic

Certops/medhallu-twins-rewritten

MedHallu twins, rewritten answers Twin pairs for medical hallucination detection. Each pair shares one PubMed abstract and one question, and holds two answers: a grounded one and a hallucinated one differing by exactly one edited span. Both answers are generated by the same model in the same call. Neither is taken from MedHallu or PubMedQA. This replaces Certops/medhallu-twins-repaired-context, where the ground-truth answer was a verbatim span of its own context in 99.5% of rows… See the full description on the dataset page: https://huggingface.co/datasets/Certops/medhallu-twins-rewritten.

sourceHugging Faceotherupdated 16d agoView on Hugging Face

1likes68downloads

Dataset Card

MedHallu twins, rewritten answers

Twin pairs for medical hallucination detection. Each pair shares one PubMed abstract and one question, and holds two answers: a grounded one and a hallucinated one differing by exactly one edited span.

Both answers are generated by the same model in the same call. Neither is taken from MedHallu or PubMedQA. This replaces `Certops/medhallu-twins-repaired-context`, where the ground-truth answer was a verbatim span of its own context in 99.5% of rows and string overlap alone separated the classes at AUC 0.9992.

Why the rewrite

PubMedQA is built by a structural split: the abstract is cut at the CONCLUSION heading, that section becomes the answer, everything before it becomes the context. So the answer is always a span of its own source document. Restore the full abstract and you restore the answer; withhold it and the answer is not entailed. There is no third setting, which is why the context could not be patched and the answers had to be replaced.

Leakage gates

Every pair passed these at generation time; failures were regenerated, not kept.

Gate	Previous dataset	This dataset
Class separation by answer/context 5-gram overlap	0.998	0.500
Style predicting which twin is hallucinated (AUC)	0.800	0.526 (chance 0.497)
Hallucinated twin is shorter	73%	20%
Within-pair word-count delta, median	-9	0

Labels come from provenance, not a judge

No model was asked whether an answer is faithful. The hallucinated twin is built by a recorded edit, and a pair is only kept if the twin rebuilds exactly from the correct answer with that span swapped. So row_type is known by construction, and edit_original_span marks precisely which text is wrong.

Splits

Split	Rows	Pairs
train	2998	1499
test	500	250

Split over pairs, never rows, so both twins stay on one side. Verified disjoint on row_id and pmid. Test is stratified by difficulty.

Columns

Column	Meaning
`row_id`	`{source_dataset}::{source_id}::{row_type}`
`question`, `context`	shared by both twins of a pair
`answer`	the generated answer for this row
`row_type`	`not_hallucinated` or `hallucinated`
`edit_original_span`	the span replaced to create the hallucination
`edit_replaced_with`	what replaced it
`edit_why_unsupported`	why the abstract does not support it
`hallucination_category`	MedHallu taxonomy, kept for comparability
`difficulty`	MedHallu difficulty label
`pmid`	PubMed id of the source abstract
`answer_containment`	5-gram containment of the answer in its context
`twin_length_delta`	word-count difference between the twins

Caveats

Constructed negatives are a narrower distribution than hallucinations seen in deployment, which are messier and mix true and false content in one sentence. A human-labelled test set is still required before any accuracy claim.

Claim-level reasoning is not generated yet.