Kavin2615/meta-openenv-hackathon-demo

AI Misuse Triage Environment

Meta PyTorch OpenEnv Hackathon Submission A text-based reinforcement learning environment for AI safety review and misuse triage.

🚀 Overview

The AI Misuse Triage Environment is a fully offline, self-contained OpenEnv-style environment where an agent plays the role of an AI safety reviewer. Given a tuple of (user_prompt, assistant_response, context), the agent must detect misuse risks, classify them, and choose appropriate mitigation actions.

This project implements a reinforcement-learning-style triage system that improves its policy via user feedback, simulating the real-world task of automated content policy enforcement.

💡 Why This Matters

As AI assistants become more capable, the risk of misuse grows proportionally:

• —Phishing campaigns drafted by LLMs are harder to distinguish from legitimate emails.
• —Prompt injection attacks can subvert AI pipelines silently.
• —Scam and harassment content can be generated at scale.

This environment formalizes misuse triage as a decision-making task, allowing researchers to train and evaluate safety agents systematically while studying the tradeoffs between false positives and false negatives.

🛠 Features

• —Operating Modes:
• —Evaluation: Deterministic rule-based triage for stable baseline testing.
• —Training: Interactive RL mode where you can provide reward signals (+1/-1) to update the agent's weight-based policy.
• —Rich Aesthetics: State-of-the-art Web UI with real-time status indicators, JSON inspection, and a premium Glassmorphism design.
• —Persistence: Agent state persists in agent_memory.json, with episodic logs stored in training_log.jsonl.
• —Automated Evaluation: A CLI interface (inference.py --minimal) for zero-manual-step grading compliant with the OpenEnv protocol.

📦 Setup & Installation

Requirements: Python 3.10+

git clone https://github.com/Kavin-2602/Ai-misuse-Triage.git
cd Ai-misuse-Triage

# (Optional) create a virtual environment
python -m venv venv
# Linux/Mac: source venv/bin/activate | Windows: venv\Scripts\activate

pip install -r requirements.txt

🕹 Running the Demo

Web Interface (Recommended)

Run the Flask app:

python app.py

Visit http://127.0.0.1:7860 to view the app.

CLI Baseline

Run the rule-based sample agent through the environment loop:

• —Full Benchmark: python inference.py
• —Single Episode: python inference.py --single
• —Automated Mode: python inference.py --minimal (JSON output only)

🧪 How it Works

The environment follows the standard OpenEnv / Gymnasium reset-step interface:

from openenv_misuse_triage import MisuseTriageEnv

env = MisuseTriageEnv(shuffle=True, seed=42)
obs, info = env.reset()

while True:
    action = my_agent.decide(obs)  # Agent produces decision
    obs, reward, terminated, _, info = env.step(action)
    if terminated: break

Output Schema

The agent must return a JSON object with these four keys:

🎯 Scoring Rubric

Scoring is fully deterministic based on a weighted rubric:

🐳 Docker & Hugging Face Deployment

Running with Docker

1. 1.Build: docker build -t ai-misuse-triage .
2. 2.Run UI: docker run -p 7860:7860 ai-misuse-triage
3. 3.Run Eval: docker run ai-misuse-triage python inference.py --minimal

Hugging Face Spaces

This project is configured for Hugging Face Spaces with a Flask-based app:

• —The Dockerfile exposes port 7860.
• —The container starts gunicorn to serve the app.
• —Push this repository to your Space to deploy.

📂 Project Structure

├── openenv_misuse_triage/ # Core Environment SDK
├── templates/             # UI Templates
├── static/                # UI Assets & JS
├── inference.py           # Evaluation Entrypoint
├── learning.py            # RL Agent Logic
├── app.py                 # Flask App Source
├── Dockerfile             # Container configuration
├── openenv.yaml           # OpenEnv metadata
├── requirements.txt       # Dependencies
└── README.md              # Documentation

Created for the Meta PyTorch OpenEnv Hackathon.