VPTT/medtriage-openenv

MedTriage OpenEnv

MedTriage is a real-world OpenEnv benchmark for emergency department decision-making. Instead of one-shot classification, agents must work through short clinical workflows: gather missing history, order high-yield diagnostics, stabilize unstable patients, and only then finalize triage, diagnosis, treatment, or disposition.

Why It Stands Out

• —Multi-step episodes instead of single-turn scoring
• —Dynamic patient evolution when urgent care is delayed
• —Resource and time pressure through test costs and deterioration penalties
• —Deterministic rubric-based graders with partial credit and safety penalties
• —Connected ED skill ladder: triage, diagnosis, treatment, and disposition

Tasks

1. vital-triage

The agent assigns an Emergency Severity Index after focused questioning, rapid tests, and early stabilization.

2. differential-diagnosis

The agent gathers evidence, identifies red flags, and submits a ranked differential diagnosis.

3. treatment-safety

The agent stabilizes the patient and proposes a safe treatment plan while avoiding allergies, contraindications, and sequencing mistakes.

4. disposition-planning

The agent chooses discharge, observation, ward, or ICU disposition with follow-up actions and rationale.

Action Space

The environment uses a unified typed action model:

class MedTriageAction(BaseModel):
    action_type: str
    question: Optional[str]
    requested_tests: List[str]
    stabilization_actions: List[str]

    esi_level: Optional[int]
    triage_reason: Optional[str]

    diagnoses: List[str]
    red_flags: List[str]
    recommended_tests: List[str]

    diagnosis: Optional[str]
    drug_name: Optional[str]
    dose_mg: Optional[float]
    route: Optional[str]
    rationale: Optional[str]

    disposition: Optional[str]
    disposition_reason: Optional[str]
    follow_up_actions: List[str]

Observation Space

class MedTriageObservation(BaseModel):
    patient: PatientInfo
    current_task: str
    task_instruction: str
    step_number: int
    max_steps: int
    last_action_result: str
    last_action_error: Optional[str]
    cumulative_reward: float
    progress: float
    done: bool
    context: Dict[str, Any]

context includes available actions, available tests, stabilization options, elapsed minutes, resource spend, revealed tests, and the required final action.

Reward Design

Rewards are dense and deterministic.

• —Focused questions and high-yield tests earn partial credit
• —Urgent stabilization earns reward and can improve visible vitals
• —Delays and unnecessary actions incur time/resource penalties
• —Final graders return normalized scores in [0.0, 1.0]
• —Unsafe treatment choices can immediately score 0.0

Clinical Realism Features

• —Hemodynamic deterioration for unstable patients if urgent actions are delayed
• —Hidden diagnostic findings that appear only after appropriate tests are ordered
• —Safety-critical scenarios including hypokalemic DKA, septic shock with CKD, NSAID-triggered asthma, transient neurologic deficit, and shock-level sepsis disposition
• —Deterministic edge cases designed to separate strong models from superficial pattern matching

Setup

pip install -e .
pip install -e ".[inference]"

Run Locally

uvicorn medtriage_env.server.app:app --host 0.0.0.0 --port 7860

Docker

docker build -t medtriage-env .
docker run -p 7860:7860 medtriage-env

Baseline Inference

Required environment variables:

• —API_BASE_URL
• —MODEL_NAME
• —HF_TOKEN
• —MEDTRIAGE_URL

Run:

python inference.py

The script emits only the required validator-safe lines:

[START] task=<task_name> env=medtriage-env model=<model_name>
[STEP] step=<n> action=<action_json> reward=<0.00> done=<true|false> error=<msg|null>
[END] success=<true|false> steps=<n> score=<score> rewards=<r1,r2,...>

Pre-Submission Validation

With the server running locally:

python validate_local.py

This checks:

• —health and task endpoints
• —normalized score ranges
• —all four tasks can reset and step
• —baseline inference exits cleanly
• —inference stdout contains only [START], [STEP], and [END] lines

Safety Note

All cases are synthetic and intended only for benchmarking and research on agent reasoning. This project is not a clinical decision support tool and must not be used for real patient care.