anon-meddial-2026/meddialbench

MedDialBench

A controlled factorial benchmark for evaluating LLM diagnostic robustness under parametric adversarial patient behaviors.

Anonymous submission to NeurIPS 2026 Datasets and Benchmarks Track. The companion paper is currently under double-blind review. Author identities and institutional affiliations are intentionally omitted. After publication, this repository will be transferred to a permanent (de-anonymized) location and this README updated accordingly.

Overview

MedDialBench evaluates how five frontier LLMs (acting as doctor agents) cope with five independently controllable adversarial patient behaviors during diagnostic consultation:

• —Logic Consistency — fabrication and contradictions
• —Health Cognition — denial and misattribution
• —Expression Style — incoherent or vague speech
• —Disclosure — withholding of relevant information
• —Attitude — dominant or impatient demeanor

Each dimension has graded severity levels (baseline / moderate / extreme), and configurations include single-dimension perturbations as well as multi-dimension combinations. The companion paper analyzes single-dimension dose-response, cross-dimension interaction (super-additivity), and per-model vulnerability profiles.

Headline numbers. Across 7,225+ pre-computed dialogues:

• —Information pollution (fabrication) produces 1.7--3.4× larger accuracy drops than information deficit (withholding).
• —Fabrication is the only dimension that significantly degrades all five models.
• —Fabrication is the sole driver of super-additive interaction (35--44% of eligible cases fail under combinations despite succeeding under each dimension alone).

Repository Contents

.
├── cases_filtered_85.json         # 85 OSCE-style clinical vignettes used in the experiments
├── claude-opus-4-6/               # pre-computed dialogues with Claude Opus 4.6 as doctor
├── qwen3.5-plus/                  # ... Qwen 3.5 Plus
├── deepseek-v3.2/                 # ... DeepSeek V3.2
├── gemini-3.1-pro/                # ... Gemini 3.1 Pro
├── gpt-5.4/                       # ... GPT-5.4
├── judge_validation/              # 201 human-annotated dialogues used to select the LLM judge
└── adherence/                     # patient-agent behavioral adherence audit (final v3)

Each model directory contains 17+ subdirectories — one per behavioral configuration (baseline, attitude=dominant, cognition=complete_denial, disclosure=withholding+logic=fabricating, …) — plus a summary.json per model and a small README.md documenting which experimental round was used per configuration.

Note on case counts. The baseline configuration covers all 107 OSCE cases (the full set used for the diagnosability filter described in the paper, §3.3); the diagnosable subset of 85 cases was retained for the 16 non-baseline configurations. The paper's main analysis (7,225 dialogues = 85 × 17 × 5) is therefore drawn from the 85-case subset, and cases_filtered_85.json lists exactly which case IDs constitute the canonical evaluation set. The repository also retains some additional dialogue files from earlier experimental rounds with intermediate filter sets; these are kept for transparency and are not included in the paper's main reported numbers.

Schema

cases_filtered_85.json

A JSON list of 85 clinical vignettes. Each entry includes:

A subset of these vignettes is derived from the OSCE cases compiled by AgentClinic (Schmidgall et al., 2024); see Licensing below.

<doctor_model>/<config>/dial_<case_id>.json

One file per (doctor model × configuration × case). Top-level fields:

judge_validation/

201 dialogues with human-annotated ground-truth labels for correct (diagnosis correctness) and misled (whether false patient information caused diagnostic error). Used to select the primary LLM judge during benchmark construction (κ = 0.882 on semantic accuracy).

adherence/

Three JSON files with the final adherence-audit results (95.5% / 92.3% / 100% rates for single-dim / dual-dim / isolation respectively, per the companion paper).

How to Reproduce the Paper's Headline Numbers

The companion paper's analysis scripts read directly from this directory tree (no API keys required). After downloading and unpacking, point the RESULTS_DIR variable in the analysis scripts at this directory and re-run them. See the companion repository (linked from the paper) for the scripts.

Licensing

• —All JSON files in this repository: CC BY 4.0.
• —A subset of the clinical vignettes in cases_filtered_85.json is derived from MedQA-style OSCE cases compiled by AgentClinic (https://github.com/SamuelSchmidgall/AgentClinic), distributed under the MIT License. The derived cases have been rewritten into colloquial patient language and augmented with behavioral configuration metadata. Users redistributing the cases must comply with MedQA's upstream license for the unrewritten portions.
• —Patient-side dialogue utterances were generated by Anthropic's Claude Opus 4.5. Use for research and evaluation is covered by Anthropic's published commercial terms at the time of generation.

No Clinical Use

These cases are fictional or rewritten research artifacts. They must NOT be used to inform patient care, clinical decision-making, or any safety-critical application.

Citation (during review)

@inproceedings{meddialbench2026,
  title     = {MedDialBench: A Controlled Factorial Benchmark for Evaluating LLM Diagnostic Robustness Under Adversarial Patient Behaviors},
  author    = {Anonymous},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track},
  year      = {2026},
  note      = {Under review}
}

A permanent citation will replace this entry after publication.