Datasetpublic

BlueZeros/AgentEHR-Bench

AGENTEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization Paper | Code AGENTEHR is a novel benchmark designed to bridge the gap between idealized experimental settings and realistic clinical environments. Unlike previous tasks that focus on factual retrieval, AGENTEHR challenges agents to perform complex clinical decision-making—such as diagnosis and treatment planning—directly within raw, high-noise EHR databases. 💡 Key Features… See the full description on the dataset page: https://huggingface.co/datasets/BlueZeros/AgentEHR-Bench.

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face

0likes1.8kdownloads

Dataset Card

AGENTEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization

**Paper** | **Code**

AGENTEHR is a novel benchmark designed to bridge the gap between idealized experimental settings and realistic clinical environments. Unlike previous tasks that focus on factual retrieval, AGENTEHR challenges agents to perform complex clinical decision-making—such as diagnosis and treatment planning—directly within raw, high-noise EHR databases.

💡 Key Features

—Realistic Clinical Benchmark: Covers six core tasks (Diagnoses, Labevents, Microbiology, Prescriptions, Procedures, and Transfers) spanning the entire patient hospitalization lifecycle.
—Toolbox MCP Server: A standardized interface providing agents access to over 19 specialized tools, including SQL execution, temporal filtering, and semantic search.
—Retrospective Reasoning: Supports the evaluation of frameworks that re-evaluate interaction history to capture latent correlations and ensure logical coherence.
—Experience Memory Bank: Facilitates strategies that crystallize successful approaches into an external memory bank.

📊 Benchmark Structure

AGENTEHR is organized into three experimental subsets based on MIMIC-IV and MIMIC-III to evaluate generalization and robustness:

Subset	Distribution Type	Description
MIMIC-IV-Common	In-Distribution	Primary benchmark assessing standard clinical reasoning capabilities on prevalent conditions.
MIMIC-IV-Rare	Label-Shift OOD	Evaluates the agent's ability to handle low-prevalence diseases where parametric knowledge is weaker.
MIMIC-III	Systemic-Shift OOD	Presents fundamental differences in table schema and higher recording density/noise.

⚡ Quick Start

Environment Setup

To use this benchmark with the official code, follow these steps:

bash

git clone https://github.com/BlueZeros/AgentEHR.git
cd AgentEHR
pip install -r requirements.txt
pip install -U vllm

Database Preparation

Download the dataset from this repository. Copy the EHRAgentBench and MIMICIIIAgentBench folders into the ./data folder in your root directory of the cloned repository.

Citation

If you find our work helpful, please cite our paper:

bibtex

@article{liao2026agentehr,
  title={AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization},
  author={Yusheng Liao and Chuan Xuan and Yutong Cai and Lina Yang and Zhe Chen and Yanfeng Wang and Yu Wang},
  journal={arXiv preprint arXiv:2601.13918},
  year={2026}
}