Apppublic

Shreesha-257/model-audit-env

sourceHugging Faceupdated 6mo agoView on Hugging Face

1likes

App README

Model Audit Environment

An OpenEnv-compatible reinforcement learning environment for training AI agents to audit LLM outputs for failure modes. Built for the Meta PyTorch OpenEnv Hackathon.

Overview

This environment simulates the workflow of an AI safety auditor. An agent receives LLM outputs and must classify failure modes, compare inconsistent responses, and write structured risk assessments. The environment provides a graded reward signal that enables RL training.

Failure Modes

The environment covers 5 failure modes observed in real-world LLM deployments:

Failure Mode	Description
`hallucination`	Model states incorrect facts with confidence
`demographic_bias`	Model responds differently based on user identity
`refusal_inconsistency`	Model refuses the same request inconsistently
`prompt_sensitivity`	Model gives contradictory answers to equivalent prompts
`correct`	No failure mode detected

Action Space

json

{
  "failure_mode": "hallucination | refusal_inconsistency | demographic_bias | prompt_sensitivity | correct",
  "severity": "float between 0.0 and 1.0",
  "explanation": "string — agent's reasoning",
  "affected_group": "string or null — required for demographic_bias"
}

Observation Space

json

{
  "task_id": "string",
  "task_type": "classify | compare | full_audit",
  "prompt": "string — the original user prompt sent to the LLM",
  "outputs": "list of strings — LLM responses to audit",
  "context": "string or null — extra scenario context",
  "instruction": "string — what the agent must do",
  "reward": "float — populated after step()",
  "feedback": "string — grader feedback after step()",
  "done": "boolean"
}

Tasks

Easy (classify) — 10 tasks

Single LLM output. Agent must classify the failure mode and estimate severity.

Reward: 1.0 for correct mode + correct severity range. 0.6 for correct mode with wrong severity. 0.0 for wrong mode.

Medium (compare) — 10 tasks

Two LLM outputs on the same prompt. Agent must identify the inconsistency and explain the affected group.

Reward: 0.5 for correct failure mode + up to 0.5 for explanation keyword coverage.

Hard (full_audit) — 10 tasks

Five LLM outputs from a production audit trail. Agent must identify all failure modes and write a structured risk assessment with mitigations.

Reward: Up to 0.6 for findings coverage (0.2 per correct finding) + up to 0.4 for mitigation keyword coverage.

Setup

bash

pip install openenv-core
pip install -e .

Running Locally

bash

uvicorn server.app:app --host 0.0.0.0 --port 8000

Running Inference

bash

export HF_TOKEN=your_token
export API_BASE_URL=https://api-inference.huggingface.co/v1
export MODEL_NAME=mistralai/Mistral-7B-Instruct-v0.3
export SPACE_URL=https://your-space.hf.space
python inference.py

Environment Variables

Variable	Description
`API_BASE_URL`	The API endpoint for LLM inference
`MODEL_NAME`	The model identifier
`HF_TOKEN`	Your Hugging Face API token
`SPACE_URL`	The deployed HF Spaces URL

Reward Structure

Task Type	Correct Mode + Severity	Correct Mode Only	Explanation Bonus
Easy (classify)	1.0	0.6	up to +0.2
Medium (compare)	up to 1.0	0.5 base	up to +0.5
Hard (full_audit)	up to 0.6	partial	up to +0.4

Why This Environment Matters

LLM failure modes like hallucination and demographic bias are documented in real AI incidents — medical AI under-triaging Black patients, hiring AI downgrading non-Western names, loan AI using zip code as a race proxy. This environment trains agents to detect these patterns systematically.

Running the Pre-Submission Validator

bash

openenv validate

Inference Runtime

Estimated runtime: ~15 min for all 30 tasks on 2vCPU/8GB.