EliasHossain/carebench
CareBench CareBench is a benchmark for process-aware evaluation of clinical LLM agents. Each case places an agent in a simulated patient encounter in which it must gather evidence through interaction, commit to a diagnosis, and propose management, while safety-critical red flags are tracked throughout the episode. Scoring is process-aware: in addition to diagnostic accuracy, the benchmark measures evidence coverage, premature closure, red-flag coverage, and unsafe commitments.… See the full description on the dataset page: https://huggingface.co/datasets/EliasHossain/carebench.
CareBench
CareBench is a benchmark for process-aware evaluation of clinical LLM agents. Each case places an agent in a simulated patient encounter in which it must gather evidence through interaction, commit to a diagnosis, and propose management, while safety-critical red flags are tracked throughout the episode. Scoring is process-aware: in addition to diagnostic accuracy, the benchmark measures evidence coverage, premature closure, red-flag coverage, and unsafe commitments.
The evaluation harness, agent and patient-simulator code, and scoring pipeline are available in the accompanying code repository.
Files
Schema
Each record contains a task instruction, a user_id, and a case object with the following fields:
case_id, patient_id, source_case_id, chief_complaint, patient_brief, history, symptoms, physical_exam, labs, imaging, medications, allergies, evidence_map, ground_truth_diagnosis, ground_truth_treatment, category, difficulty, language, translation_status.
The evidence_map links each ground-truth finding to the interaction step at which it becomes observable, which is what enables process-level scoring.
Source data and attribution
The 50 cases are adaptations of de-identified, published case material from two public corpora:
- 31 cases derive from PMC-Patients (Zhao et al.), released under CC-BY-NC-SA-4.0. Original PMC article identifiers are preserved in
case_id(for examplepmc_7353859-1). - 19 cases derive from MedChain, released under CC-BY-NC-4.0. These cases were translated from Chinese and curated;
translation_statusrecords the audit state.
All cases were restructured into the CareBench schema, enriched with evidence maps and red-flag annotations, and reviewed for consistency.
License
This dataset is released under CC-BY-NC-SA-4.0, the most restrictive combination compatible with both upstream licenses. Commercial use is not permitted. If you redistribute adaptations of this dataset, you must do so under the same license and preserve attribution to PMC-Patients and MedChain.
Intended use
CareBench is intended for research on the evaluation of clinical decision-making behavior in LLM agents. It is not a diagnostic tool, it must not be used to provide medical advice, and model performance on this benchmark does not certify clinical competence.
Authors and contact
CareBench is developed by Elias Hossain, Tasfia Nuzhat Ornee, Shubhashis Roy Dipta, Andrea Rivera-Sepulveda, Pegah Khosravi, and Niloofar Yousefi.
The dataset is maintained by Elias Hossain. For questions, contact elias.hossain191@gmail.com or open a discussion on this repository.
Citation
To cite the dataset itself:
@misc{hossain2026carebenchdata,
author = {Hossain, Elias and Ornee, Tasfia Nuzhat and Roy Dipta, Shubhashis and Rivera-Sepulveda, Andrea and Khosravi, Pegah and Yousefi, Niloofar},
title = {CareBench: A Process-Aware Benchmark for Clinical LLM Agents},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/EliasHossain/carebench}}
}The accompanying paper is currently under review and an arXiv preprint is coming soon. Once it is available, a paper citation entry will be added here and should be preferred when citing the benchmark methodology.