YangL1122/DetailBench

1---2language:3- en4- zh5license: cc-by-4.06task_categories:7- text-generation8- question-answering9tags:10- hallucination11- regulatory-compliance12- preference-optimization13- dpo14- long-context15- detail-faithfulness16size_categories:17- 10K<n<100K18configs:19- config_name: default20  data_files:21  - split: train22    path: data/train-*.jsonl23  - split: validation24    path: data/val-*.jsonl25  - split: test26    path: data/test-*.jsonl27---28 29# DetailBench30 31**A Benchmark for Detail Hallucination in Long Regulatory Documents**32 33DetailBench is a benchmark for evaluating and mitigating *detail hallucination* in LLM outputs on long regulatory documents.34 35## Overview36 37Large language models frequently produce *detail hallucinations*—subtle errors in threshold values, units, scopes, obligation levels, and conditions—when processing long regulatory documents. DetailBench provides:38 39- **322 source documents** (172 real + 150 synthetic) from three jurisdictions40- **13,000 preference pairs** (10,000 train / 1,000 validation / 2,000 test)41- **Five detail error types** (τ₁–τ₅) with balanced training distribution42- **Three context-length tiers**: Short (8K–16K), Medium (16K–32K), Long (32K–64K tokens)43 44## Data Sources45 46| Source | Count | Description |47|--------|------:|-------------|48| GB Standards | 65 | Chinese national standards on hydrogen production, storage, transportation, and safety |49| US CFR | 31 | Code of Federal Regulations (Title 49: Transportation, Title 40: Environmental Protection) via eCFR API |50| EUR-Lex | 76 | EU regulations on hydrogen infrastructure, clean energy, pressure equipment via CELLAR API |51| Synthetic | 150 | Domain-template generated documents for training augmentation |52 53## Schema54 55Each sample in the JSONL files contains:56 57```json58{59  "sample_id": "test_00000",60  "context_tier": "long",61  "token_count": 43368,62  "documents": [63    {64      "doc_id": "SYNTH_0075",65      "source": "synthetic",66      "segments": [67        {68          "segment_id": "SYNTH_0075_seg_0",69          "text": "...",70          "token_count": 60571        }72      ]73    }74  ],75  "query": "An electrolyser plant produces hydrogen at ...",76  "chosen": {77    "is_compliant": true,78    "constraints": [79      {"type": "tau_1", "description": "...", "value": "82", "unit": "°C"}80    ],81    "evidence": [82      {"segment_id": "...", "quote": "..."}83    ]84  },85  "rejected": {86    "is_compliant": true,87    "constraints": ["... (with one perturbed detail)"],88    "evidence": ["..."]89  },90  "perturbation": {91    "error_type": "tau_5_condition",92    "original_value": "where appropriate",93    "perturbed_value": "[dropped]",94    "detail_element_id": "...",95    "segment_id": "..."96  },97  "detail_elements": [98    {99      "element_id": "...",100      "type": "tau_1",101      "value": "3928.0",102      "unit": "kg",103      "span": [46, 55],104      "segment_id": "...",105      "quote": "..."106    }107  ]108}109```110 111## Detail Error Taxonomy112 113| Type | Name | Description | Example |114|------|------|-------------|---------|115| τ₁ | Threshold | Numeric value errors | "pressure ≤ **35** MPa" → "pressure ≤ **45** MPa" |116| τ₂ | Unit | Measurement unit errors | "distance in **meters**" → "distance in **feet**" |117| τ₃ | Scope | Applicability scope errors | "for **indoor** facilities" → "for **all** facilities" |118| τ₄ | Level | Obligation level errors | "**shall** comply" → "**should** comply" |119| τ₅ | Condition | Conditional clause errors | "if temperature **exceeds 60°C**" → condition dropped |120 121## Evaluation Metrics122 123- **Compliance Accuracy**: Fraction of correct compliance judgments124- **Detail Error Rate (DER)**: Per-type and overall error rate on detail elements125- **Evidence F1**: Precision/recall/F1 of predicted evidence citations126- **Evidence Consistency**: Fraction of citations where quoted text matches source127 128## Usage129 130```python131from datasets import load_dataset132 133ds = load_dataset("YOUR_USERNAME/DetailBench")134 135# Access splits136train = ds["train"]      # 10,000 samples137val = ds["validation"]   # 1,000 samples138test = ds["test"]        # 2,000 samples139 140# Example: inspect a test sample141sample = test[0]142print(sample["query"])143print(sample["context_tier"])  # "short", "medium", or "long"144print(len(sample["documents"]))145```146 147## Split Statistics148 149| Split | Samples | Short | Medium | Long |150|-------|--------:|------:|-------:|-----:|151| Train | 10,000 | 6,463 | 2,263 | 1,274 |152| Val | 1,000 | 605 | 249 | 146 |153| Test | 2,000 | 1,215 | 519 | 266 |154 155Error type distribution in the training set is balanced at 20% each (2,000 per type).156 157## License158 159This dataset is released under CC-BY-4.0. The underlying regulatory documents are sourced from public government repositories (eCFR, EUR-Lex, openstd.samr.gov.cn).160