sniki28/content-moderation-queue

Content Moderation Queue — OpenEnv

A real-world AI training environment that simulates the daily work of a Trust & Safety moderation team. Agents learn to triage a queue of social media posts — classifying policy violations, applying graduated consequences, handling user appeals, and escalating crisis content.

Motivation

Content moderation is one of the most consequential real-world tasks at scale. Every major social platform employs thousands of human moderators who make dozens of decisions per hour under strict policy guidelines. This environment:

• —Trains agents to apply nuanced, context-sensitive policies (not just keyword filters)
• —Rewards appropriate severity calibration with partial-credit scoring
• —Tests whether agents read author history and thread context before deciding
• —Evaluates crisis detection (self-harm signals must be escalated, not removed)

Environment Description

The agent receives a queue of social media posts and must moderate each one using a 6-action policy framework:

Observation Space

Each observation contains:

Action Space

{
  "action_type": "remove",
  "violation_type": "hate_speech",
  "reasoning": "Post contains ethnic slur with dehumanizing language"
}

• —action_type (required): One of 6 moderation actions
• —violation_type (optional): spam | hate_speech | harassment | misinformation | csam | illegal_services | doxxing | self_harm_risk | none
• —reasoning (optional): Agent explanation — logged but not used for grading

Tasks

Task 1 — Binary Content Moderation (Easy)

• —Posts: 8 | Max steps: 12
• —Actions: approve or remove only
• —Posts contain clear, unambiguous signals: obvious spam, explicit slurs, direct threats vs. cooking tips, community announcements
• —Scoring: Binary exact match — 1.0 correct, 0.0 wrong. Score = mean.
• —Expected baseline score: ~0.75

Task 2 — Tiered Policy Enforcement (Medium)

• —Posts: 10 | Max steps: 18
• —Actions: approve / warn / remove / ban_temporary / ban_permanent
• —Includes edge cases: harsh-but-legal film criticism, first vs. repeat offenders, political speech, zero-tolerance violations
• —Scoring: Action distance score (70%) + violation type identification (30%). Partial credit for being one level off.
• —Expected baseline score: ~0.55

Task 3 — Full Queue Management with Context & Appeals (Hard)

• —Posts: 12 | Max steps: 24
• —Actions: All 6 including escalate
• —Requires: reading author history (5+ violations → permanent ban), thread context (gaming slang ≠ threat), crisis detection (suicidal ideation → escalate, don't remove), appeal handling
• —Scoring: Action score (50%) + context-aware bonus (30%) + violation type (20%)
• —Expected baseline score: ~0.40

Reward Function

• —Per-step, non-sparse: every post scores independently (0.0–1.0)
• —Partial credit: being one action-level off (e.g., warn when remove is correct) scores ~0.65 instead of 0
• —Context bonus (hard task): +0.3 for posts where correct answer requires author history or thread context
• —Episode score: mean of all per-post scores

API Endpoints

Setup & Usage

Local Development

# Clone / navigate to project
cd content-moderation-env

# Install dependencies
pip install -r requirements.txt

# Start the server
uvicorn app:app --host 0.0.0.0 --port 7860 --reload

Docker

docker build -t content-moderation-env .
docker run -p 7860:7860 content-moderation-env

Run Baseline Inference

export API_BASE_URL="https://api-inference.huggingface.co/v1"
export MODEL_NAME="meta-llama/Meta-Llama-3-8B-Instruct"
export HF_TOKEN="hf_your_token_here"
export ENV_BASE_URL="http://localhost:7860"

python inference.py

Baseline Scores

Measured using meta-llama/Meta-Llama-3-8B-Instruct (temperature=0):

Scores are reproducible at temperature=0.

Project Structure

content-moderation-env/
├── openenv.yaml              # OpenEnv spec metadata
├── Dockerfile                # HF Spaces / Docker deployment
├── requirements.txt          # Python dependencies
├── inference.py              # Baseline agent script (OpenAI client)
├── app.py                    # FastAPI server (reset/step/state endpoints)
├── README.md
└── environment/
    ├── __init__.py
    ├── models.py             # Pydantic: Observation, Action, Reward, StepResult
    ├── env.py                # ContentModerationEnv class
    ├── tasks.py              # Task definitions + deterministic graders
    └── data/
        └── posts.json        # 30 labeled posts with ground truth

HF Spaces Deployment

This environment is deployed as a Hugging Face Space tagged with openenv.

The Space exposes the full OpenEnv HTTP API. Set the following secrets in your Space settings:

API_BASE_URL   # LLM endpoint
MODEL_NAME     # Model to use for inference
HF_TOKEN       # Your Hugging Face API token