KRLabsOrg/lettucedetect-prose-hallucination

LettuceDetect Prose Hallucination Dataset

Token-level hallucination annotations on LLM answers grounded in prose context, drawn from two public RAG hallucination resources and mapped into one unified taxonomy. This is the prose counterpart to the structured-context (code, tool output, documents) collection — together they let a single detector be trained across modalities.

Two sources sit side by side, distinguished by the dataset field:

PsiloQA matters because its hallucinations are not synthetically injected — they occur naturally when models answer grounded questions. That makes this collection a check that a detector generalizes beyond any single corruption process, and it spans 14 languages.

Statistics

Per source:

RAGTruth has no native validation split, so validation is PsiloQA only.

Hallucination Taxonomy

Spans are labeled with the LettuceDetect unified taxonomy — three injectable top-level categories, each with subtypes. Both sources map into the same space.

How each source is typed:

• —RAGTruth maps mechanically from its native labels (Evident/Subtle Conflict → contradiction; Baseless Info → unsupported_addition, or fabricated_reference when the span introduces a proper noun absent from the context).
• —PsiloQA ships only binary character spans with no error type, so each span is classified into the taxonomy by an LLM that reads the context and the answer with the span marked. Clean (non-hallucinated) PsiloQA answers are kept as natural negatives.

Data Format

Each line in the JSONL files is one sample:

{
  "prompt": "User request: ...\n\n...grounded passage...",
  "context": "...grounded passage...",
  "question": "the user request",
  "answer": "the LLM answer (hallucinated or clean)",
  "labels": [
    {"start": 23, "end": 41, "label": "contradiction",
     "category": "contradiction", "subcategory": "numerical"}
  ],
  "split": "train",
  "task_type": "qa",
  "dataset": "psiloqa",
  "language": "en",
  "context_modality": "prose",
  "category": "contradiction",
  "subcategory": "numerical",
  "metadata": "{\"id\": \"...\", \"is_hallucinated\": true, \"classifier_model\": \"Qwen/Qwen3.6-35B-A3B\"}"
}

Fields:

• —prompt — the full model input, request first (User request: {question}\n\n{context}) so it survives context truncation
• —context / question — the grounding passage and the request, separately (question is null for RAGTruth summarization). PsiloQA's foreign-language questions are kept verbatim.
• —answer — the LLM response (hallucinated or clean)
• —labels — annotated hallucination spans (empty for clean samples)
• —category / subcategory — unified taxonomy fields (sample-level is a majority vote over its spans)
• —dataset — source discriminator (psiloqa or ragtruth)
• —metadata — a JSON-encoded string of source-specific provenance. Parse with json.loads.

Usage

import json
from datasets import load_dataset

ds = load_dataset("KRLabsOrg/lettucedetect-prose-hallucination")

# Everything, or filter by source / language
psiloqa = ds["train"].filter(lambda r: r["dataset"] == "psiloqa")
german = ds["train"].filter(lambda r: r["language"] == "de")

for sample in ds["train"]:
    if sample["labels"]:
        meta = json.loads(sample["metadata"])
        print(sample["language"], sample["category"], meta.get("is_hallucinated"))

Limitations

PsiloQA span types are assigned by an LLM rather than human annotators, so the (category, subcategory) labels carry model noise; the underlying spans are from the original human/automatic PsiloQA annotation. RAGTruth subcategories are derived from coarse native labels and are not exhaustive.

Citation

@misc{Kovacs:2025,
      title={LettuceDetect: A Hallucination Detection Framework for RAG Applications},
      author={Ádám Kovács and Gábor Recski},
      year={2025},
      eprint={2502.17125},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.17125},
}

Please also cite the original PsiloQA and RAGTruth sources.

License

CC-BY-4.0