Datasetpublic

Certops/medhallu-twins-judged

MedHallu twins, judged (claim-level faithfulness) Splits split rows train 2,998 test 500 train is drawn from pqa_artificial; test is a disjoint set of whole twin pairs whose examples never appear in train (no twin leakage). Both are class-balanced. The repaired-context MedHallu twins run through a claim-level faithfulness judge, for training a small model to detect medical hallucinations and explain why. Each row keeps the twin (question, answer… See the full description on the dataset page: https://huggingface.co/datasets/Certops/medhallu-twins-judged.

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes88downloads

Dataset Card

MedHallu twins, judged (claim-level faithfulness)

Splits

split	rows
`train`	2,998
`test`	500

train is drawn from pqa_artificial; test is a disjoint set of whole twin pairs whose examples never appear in train (no twin leakage). Both are class-balanced.

The repaired-context MedHallu twins run through a claim-level faithfulness judge, for training a small model to detect medical hallucinations and explain why. Each row keeps the twin (question, answer, context) and adds the judge's decomposition.

Judged with gemini-3.1-flash-lite (DeepEval faithfulness, re-expressed as staged batches), grounding each claim against the full PubMed abstract (raw_context_grounding = True) rather than lossily extracted "truths". This is a partial, growing judged subset: 500 rows so far (250 not_hallucinated, 250 hallucinated).

The judge agrees with the twin's provenance on 489/500 (98%) of rows.

Columns

Twin columns (see the source dataset) plus the derived judge output:

field	meaning
`predicted`	judge label: `hallucinated` if `score < threshold` else `not_hallucinated`
`agrees`	`predicted == row_type` (True = judge matches provenance)
`score`	fraction of claims faithful to the context (0-1)
`claims`	claims extracted from the answer
`truths`	grounding text (the raw context, since `raw_context_grounding`)
`verdicts_json`	JSON list of `{claim, verdict (yes/no/idk), reason}` per claim
`score_reasoning`	natural-language justification of the score
`n_claims`,`n_yes`,`n_no`,`n_idk`	verdict composition
`has_contradiction`	any `no` verdict (a claim the context refutes)
`fully_grounded`	every claim `yes`
`judge_model`,`raw_context_grounding`,`threshold`	judge provenance

verdicts is stored as a JSON string (verdicts_json) so every row parses regardless of nesting; json.loads it for the per-claim breakdown.

Using the labels

—Contradiction-style hallucinations: filter has_contradiction.
—Any-unsupported labelling: use predicted (the score < threshold rule).
—Clean training rows: filter agrees to keep only where the judge and the planted provenance agree.

Attribution

Built on MedHallu (arXiv:2502.14302) and PubMedQA (Jin et al., EMNLP 2019); context from PubMed abstracts (NCBI E-utilities). Judge: DeepEval faithfulness via gemini-3.1-flash-lite. Cite MedHallu and PubMedQA.