baggybro/meta-pytorch-openenv

AI Content Moderation & Forensic Fact-Checker (OpenEnv)

 

Overview & Motivation

Modern social platforms manage content moderation through a suite of forensic signals, rather than relying on binary classification. These signals include AI-manipulation detectors, source credibility scores, real-time virality trackers, and third-party fact-check APIs.

This environment models the complete decision workflow as a Reinforcement Learning episode:

• —The agent functions as an AI Forensic Fact-Checker interacting with a moderation dashboard.
• —Each operational step presents a new content item paired with rich forensic metadata.
• —The agent must synthesize multiple technical signals to select the appropriate policy action.
• —The environment provides dense rewards, granting partial credit per step rather than solely at the episode terminus.

This framework directly evaluates whether an LLM agent can reason about conflicting probabilistic signals and apply policy-aligned decisions—a capability gap absent in standard benchmarks.

Environment Specification

Action Space

The agent selects one of three strategic moderation actions per content item:

Observation Space

Each step supplies a structured forensic context:

Forensic Metadata Signals:

Reward Function

The ModerationGrader algorithm produces a bounded score in the domain [0.0, 1.0]. The reward function evaluates four criteria:

Penalties:

• —False Positive (flagging accurate content): -0.3
• —Critical Miss (allowing fabricated content): -0.5

Task Descriptions

1. easy_misinfo (Easy)

Dataset: 4 text-based items. Objective: Resolve straightforward cases such as demonstrable scientific errors versus verified encyclopedic facts where all metadata signals are in consensus.

2. medium_misinfo (Medium)

Dataset: 6 mixed-media items. Objective: Triage AI-manipulated media and deceptive health claims containing conflicting signals. Requires weighted reasoning between manipulation_likelihood and fact_check_confidence.

3. hard_misinfo (Hard)

Dataset: 6 mixed-media items. Objective: Manage national security deepfakes and severe public safety risks. The agent must recognize that low fact-check confidence combined with high consequence mandates an escalate action, despite lacking definitive verification.

4. expert_misinfo (Expert)

Dataset: 4 heavily adversarial instances. Objective: Navigate advanced context collapse, rhetorical sarcasm, false authority traps, and simulated zero-day cyber threats. Designed to explicitly test the maximum contextual understanding bounds of state-of-the-art models.

Baseline Evaluation

Baseline Model: Qwen/Qwen2.5-72B-Instruct (HuggingFace Inference Router)

Technical Setup

1. Configuration

Copy the example environment file and populate the necessary authentication keys:

cp .env.example .env

2. Execution

Execute the full baseline inference script:

python inference.py

3. REST API Server

Initialize the local development server:

uvicorn moderation_env.server:app --host 0.0.0.0 --port 8000

Valid endpoint requests:

# Verify connection
curl http://localhost:8000/

# Initialize target trajectory
curl -X POST "http://localhost:8000/reset?task=easy_misinfo"

# Execute step transition
curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"decision": "flag", "reasoning": "High fact_check_confidence of 0.95 confirms fabrication."}'

Containerization

Construct and initialize the Docker container:

docker build -t moderation-env .
docker run -p 8000:8000 \
  -e HF_TOKEN=your_token \
  -e API_BASE_URL=https://router.huggingface.co/v1 \
  -e MODEL_NAME=Qwen/Qwen2.5-72B-Instruct \
  moderation-env

System Architecture

meta-pytorch-openenv/
├── moderation_env/
│   ├── env.py        # Environment integration layer
│   ├── graders.py    # Deterministic scoring protocol
│   ├── models.py     # Pydantic data schemas
│   ├── server.py     # FastAPI server endpoints
│   └── tasks.py      # Dataset repository
├── inference.py      # Automated baseline evaluation script
├── verify_submission.py  # Local validation pipeline
├── openenv.yaml      # OpenEnv deployment metadata
└── Dockerfile        # Container specifications