utk7rsh/clinical-prior-authorization

Clinical Prior Authorization — OpenEnv

An OpenEnv-compliant healthcare workflow environment where an AI agent completes a three-step prior authorization case. The agent must determine eligibility, match clinical evidence to policy criteria, and draft a medical-necessity appeal letter for a denial.

What the environment does

Each episode uses one randomly selected clinical case from 13 diverse scenarios. The case moves through three tasks in order:

1. 1.eligibility_check
2. 2.policy_match
3. 3.appeal_draft

The environment exposes these endpoints:

• —POST /reset — start a new episode
• —POST /step — submit one action for the current task
• —GET /state — inspect the current episode state
• —GET /tasks — list task definitions

The root endpoint / serves a health check and metadata.

All Endpoints

Task structure

1) Eligibility check — easy

Decide whether the patient has active coverage and whether the procedure is eligible for coverage.

Action fields:

• —task_type
• —eligibility_decision
• —coverage_reason

2) Policy match — medium

Identify which policy criteria are supported by the chart and which are not.

Action fields:

• —task_type
• —met_criteria
• —unmet_criteria
• —confidence (optional bonus)

3) Appeal draft — hard

Write a formal medical-necessity appeal letter for the denial.

Action fields:

• —task_type
• —appeal_letter
• —appeal_grounds

Observation structure

The observation includes:

• —task
• —patient
• —procedure
• —clinical_notes
• —policy_criteria
• —previous_denials
• —step_count
• —done
• —info

Scoring

Each task returns a score in the range [0.0, 1.0].

Eligibility check

• —10% format valid (action submitted with correct task_type)
• —50% correct decision
• —40% reasoning quality

Policy match

• —60% met-criteria F1
• —40% unmet-criteria recall
• —5% optional confidence bonus, capped at 1.0

Appeal draft

• —20% letter structure
• —30% clinical accuracy
• —25% appeal grounds
• —15% medical-necessity language
• —10% explicit denial handling

The graders are deterministic for a given action and scenario.

Reward Breakdown

Every /step response now includes a reward_breakdown dict with named sub-components:

{
  "reward": 0.85,
  "reward_breakdown": {
    "format_valid": 0.10,
    "decision_correct": 0.50,
    "reasoning_keywords": 0.25,
    "keywords_found": ["eligible", "criteria", "m17.11"]
  }
}

Scenario Coverage (13 Cases)

Difficulty Selection

Pass difficulty to /reset to control which scenario pool is used:

• —easy — Scenarios 1, 8, 13 (all criteria clearly met)
• —medium — Scenarios 2, 6, 12 (one gap, quantity issue, or expired PA)
• —hard — Scenarios 3, 4, 5, 7, 9, 10, 11 (inactive insurance, prior denial, OON, experimental, step therapy, BMI, plan exclusion)
• —(default) — Random from all 13

Why This Environment is Hard for LLMs

1. 1.Policy-evidence gap: The agent must cross-reference clinical notes against structured policy criteria — not just summarize text.
2. 2.Eligibility traps: Inactive insurance (is_active: false) and out-of-network flags (network_status: out_of_network) are subtle details buried in structured fields, not narrative text.
3. 3.Quantity limits: Catching "second procedure this year" requires counting, not just comprehension.
4. 4.Experimental exceptions: Scenario 7 requires understanding that needs_review is the correct answer — neither approve nor deny.
5. 5.Appeal letter quality: The grader evaluates both form (structure, formality) and substance (clinical accuracy, medical necessity language).

Concurrent Session Support

The environment now supports multiple concurrent agents via UUID-keyed sessions:

# Each agent gets its own session_id
POST /reset → {"session_id": "abc-123", "observation": {...}}

# Use session_id in subsequent calls
POST /step?session_id=abc-123  →  {...}
GET  /state?session_id=abc-123 →  {...}
POST /explain  {"session_id": "abc-123", "task": "eligibility_check"}

Baseline Performance

Evaluated using gpt-4o-mini (temperature=0.1) over 4 scenarios × 3 runs each.

Note: Graders are deterministic for a fixed (action, scenario) pair. Score variance across runs is entirely due to LLM output stochasticity at temperature=0.1. Results will differ slightly with different models or temperatures.

Baseline inference script requirements

The repository must include a root-level inference.py.

The script must:

• —use the OpenAI client for all LLM calls
• —read API_BASE_URL, MODEL_NAME, and HF_TOKEN from the environment
• —run against the submitted environment
• —emit exactly these stdout line types, in this order:

[START] task=<task_name> env=<benchmark> model=<model_name>
[STEP] step=<n> action=<action_str> reward=<0.00> done=<true|false> error=<msg|null>
[END] success=<true|false> steps=<n> score=<score> rewards=<r1,r2,...,rn>

Rules:

• —one [START] line at episode begin
• —one [STEP] line immediately after each env.step()
• —one [END] line at episode end, always emitted (even on exception)
• —reward and rewards must be formatted to 2 decimal places
• —booleans must be lowercase
• —error must be the raw last-action error string or null

Known Limitations

• —Single-session state (legacy): The v1.0 environment held one active episode at a time. This is fixed in v1.1 — concurrent sessions are fully supported via UUID-keyed session IDs.
• —In-memory only: All state (sessions, metrics, leaderboard) is stored in-memory and resets when the container restarts. This is by design — the OpenEnv spec expects Docker-isolated evaluation runs.
• —Synthetic denial injection: Scenarios without a prior denial have one injected automatically for the appeal_draft task. This is documented and intentional.

File structure

.
├── app.py
├── inference.py
├── openenv.yaml
├── Dockerfile
├── .dockerignore
├── requirements.txt
└── README.md

Local run

Docker

docker build -t clinical-prior-auth .
docker run -p 7860:7860 clinical-prior-auth

Python

pip install -r requirements.txt
python app.py

Running the baseline agent

export API_BASE_URL=https://api.openai.com/v1
export MODEL_NAME=gpt-4o-mini
export OPENAI_API_KEY=sk-...          # primary credential (OpenEnv spec)
# export HF_TOKEN=hf_...             # fallback for HF-hosted model endpoints
export ENV_URL=http://localhost:7860
python inference.py

Validation checklist

Before submission, verify that:

• —the Space responds on /reset
• —the Docker image builds successfully
• —openenv validate passes
• —inference.py is present at the repo root
• —the stdout contract matches the required log format exactly
• —GET /metrics returns aggregate statistics
• —POST /explain returns component-level score breakdowns

License

Apache 2.0