utk7rsh/Arbiter_Gen1

<div align="center">

⚖️ ARBITER

Adversarial Bias Investigation Through Evidence-based Reinforcement Training

A 7-Level RL Environment for Training LLM Agents to Conduct Causal Audits of AI Decision Systems

   

Teaching AI to investigate other AI — not just detecting bias, but proving causation.

🚀 Live Environment · 📝 Blog Post · 📊 Pitch Deck · 🧪 Training Notebook

</div>

🎯 The Problem

AI systems make high-stakes decisions about people every day — loan approvals, hiring, insurance claims, criminal risk assessment. When these systems discriminate, the standard response is to run a statistical fairness test: "Does the denial rate differ across demographic groups?"

But correlation is not causation.

A 40% higher denial rate for a specific zip-code cluster could be:

Wrong diagnosis → wrong fix → continued harm or unnecessary intervention.

Today's auditing tools can tell you that a disparity exists. ARBITER trains an AI agent to discover why — conducting a full causal investigation that identifies the chain from proxy feature to hidden mediator to outcome, and producing an evidence-backed audit report.

💡 What is ARBITER?

ARBITER is an OpenAI Gym-compatible reinforcement learning environment that trains LLM agents to perform end-to-end causal audits of AI decision systems. It is the first environment that combines:

• —🔍 Causal investigation — agents query records, analyze feature distributions, and run counterfactual interventions on a synthetic causal DAG
• —🧠 Dense multi-component rewards — 8 reward signals per episode (not just terminal), making RL training ~15× more sample-efficient
• —🎓 Auto-advancing curriculum — 7 difficulty levels from basic auditing to adversarial multi-agent scenarios
• —⚔️ Adversarial dynamics — the system under audit actively fights back with obfuscation tactics
• —🌐 Domain generalization — a single LLM call generates complete domain configs for any AI decision domain
• —👥 Multi-agent coordination — two auditors investigate the same case with shared belief states

🏗️ Architecture

arbiter/
├── env/                          # Core environment
│   ├── environment.py            # Main ArbiterEnv (reset/step/render) — 526 lines
│   ├── graph.py                  # Causal DAG generator (NetworkX) — 3 anomaly types
│   ├── reward.py                 # 8-component dense reward function
│   ├── counterfactual.py         # Do-calculus style causal interventions
│   ├── claims.py                 # 3 claim types with per-field verification
│   ├── decoys.py                 # 2 domain-aware decoy generators per episode
│   ├── defender.py               # Rule-based + frequency-adaptive obfuscation engine
│   ├── schema_drift.py           # Level 6: Mid-episode regulatory framework changes
│   ├── dual_env.py               # Level 7: Dual-auditor co-investigation
│   ├── meta_overseer.py          # Logical consistency checker (4 contradiction rules)
│   ├── curriculum.py             # Auto-advancing 7-level curriculum
│   ├── domain_config.py          # Pydantic schema for domain configurations
│   ├── groq_generator.py         # LLM-powered domain config generator
│   ├── openenv_wrapper.py        # OpenEnv hackathon compatibility layer
│   └── rubrics.py                # Evaluation rubrics
├── training/
│   ├── sft_generator.py          # Expert demonstration trajectory generator
│   └── grpo_trainer.py           # Dense-reward GRPO with KL-regularised updates
├── server.py                     # FastAPI backend for the research dashboard
└── demo/                         # Interactive demo scripts

🔬 The Environment in Detail

Observation Space

Each episode presents the agent with a complete AI decision scenario:

Action Space — 8 Structured Actions

The agent outputs JSON-structured actions, divided into three categories:

🔎 Investigation Actions (gather evidence):

{"type": "QUERY_RECORDS", "feature_filter": {"income": ">50000"}, "outcome_filter": "denied"}
{"type": "QUERY_FEATURE_DISTRIBUTION", "feature_id": "zip_code_cluster", "group_by": "demographic"}
{"type": "QUERY_COUNTERFACTUAL", "record_id": "rec_0012", "feature_id": "zip_code_cluster", "counterfactual_value": "cluster_3"}

📋 Claim Actions (build the audit report):

{"type": "CLAIM_CAUSAL", "claim": {"cause_feature": "zip_code_cluster", "effect_outcome": "denial_rate_overall", "mechanism": "internal_risk_score", "direction": "positive", "confidence": "HIGH", "anomaly_type": "proxy_discrimination"}}
{"type": "CLAIM_COUNTERFACTUAL", "claim": {"subject_record": "rec_0012", "counterfactual_feature": "zip_code_cluster", "predicted_outcome_change": "approved"}}
{"type": "CLAIM_THEORY_OF_MIND", "claim": {"defender_action": "obfuscating", "obfuscation_method": "proxy_laundering", "target_link": "zip_code_cluster→internal_risk_score"}}

🧭 Navigation Actions (manage the investigation):

{"type": "FLAG_HYPOTHESIS", "hypothesis_type": "proxy_discrimination", "status": "ACTIVE"}
{"type": "SUBMIT_REPORT", "anomaly_type": "proxy_discrimination", "affected_demographic": "zip_code_cluster_7", "recommended_action": "remove_proxy_pathway"}

Dense Reward Function — 8 Components

Unlike typical RL environments with a single terminal reward, ARBITER provides ~15 gradient-relevant signals per 20-step episode:

Penalties: Consistency violations (−1.0 each from the Meta-Overseer), HIGH-confidence wrong claims (−0.5), budget inefficiency deductions.

Key insight: Dense intermediate rewards make GRPO training ~15× more sample-efficient than terminal-only environments, enabling a 1.5B parameter model to learn complex causal reasoning in ~150 episodes.

🎓 7-Level Curriculum

ARBITER implements an auto-advancing curriculum — the agent progresses when its rolling mean reward over 30 episodes exceeds the level threshold. No manual intervention required.

Level 1 ─── Basic Auditing ──────────────── Threshold: 20.0
   │         Single anomaly type, learn to query → claim → submit
   ▼
Level 2 ─── Decoy Navigation ───────────── Threshold: 18.0
   │         +2 decoys per episode; must investigate AND eliminate
   ▼
Level 3 ─── Full Complexity ─────────────── Threshold: 15.0
   │         All 3 anomaly types + counterfactual queries unlocked
   ▼
Level 4 ─── Adversarial Defender ────────── Threshold: 15.0
   │         System fights back: link substitution, record injection,
   │         proxy laundering (3 obfuscation actions/episode)
   │         Theory of Mind claims unlock (+3.0 reward)
   ▼
Level 5 ─── Adaptive Defender ──────────── Threshold: 15.0
   │         Defender learns which tactics fool the agent and
   │         preferentially uses them (frequency-adaptive strategy)
   ▼
Level 6 ─── Schema Drift ──────────────── Threshold: 15.0
   │         Mid-episode: regulatory framework changes — a compliant
   │         feature becomes non-compliant. Agent must DETECT the
   │         change within 4 steps. New action: FLAG_SCHEMA_CHANGE
   │         Correct detection: +4.0 | Missing it: −2.0
   ▼
Level 7 ─── Dual-Auditor Network ──────── Threshold: 12.0
             Two LLM agents investigate the same case simultaneously.
             One is secretly biased toward Type-1 diagnoses.
             Modes: Collaborative (50/50) or Competitive (70/30)
             New actions: BROADCAST_CLAIM, CHALLENGE_PARTNER
             Partner bias detection: +3.0 | Wrong challenge: −1.0

📊 Results

Training Performance Across Levels

All results from GRPO training with Qwen 2.5 1.5B Instruct + LoRA, batch size 8, KL coefficient 0.05:

Behavioural Emergence

The agent develops increasingly sophisticated investigation strategies over training:

                          Early Training        Late Training
                          ─────────────         ─────────────
FALLBACK (invalid)            9–18                  1–2
CLAIM_CAUSAL                  14–19                 25–28
CLAIM_COUNTERFACTUAL           0–6                  14–18
CLAIM_THEORY_OF_MIND           0–4                  11–15
QUERY_COUNTERFACTUAL           7                     17
Evidence Chain Bonus          1.0                   3.9

Three key learning signals:

1. 1.Investigation sophistication — the agent progresses from basic record queries to counterfactual interventions, mirroring the progression of a human auditor
2. 2.Evidence discipline — the chain bonus triples, showing the agent learns to cite specific query results in its claims rather than guessing
3. 3.Calibrated confidence — HIGH-confidence wrong claims decrease over training, demonstrating learned epistemic humility

Training Stability

• —KL divergence between the policy and SFT reference model remains stable at 0.007–0.033 throughout all training runs — no catastrophic drift
• —Policy log-prob steadily increases from −3.34 to −2.78, indicating the model becomes more confident in its learned investigation strategy
• —GRPO loss trends negative (−0.42 → −0.96), confirming consistent policy improvement

🌐 Domain Generalisation

ARBITER is not locked to a single domain. The GroqGraphGenerator takes a plain-English description and produces a complete, validated DomainConfig:

from arbiter.env.groq_generator import GroqGraphGenerator

gen = GroqGraphGenerator()

# Generate configs for any AI decision domain
loan_config    = gen.generate_cached("A bank's automated loan approval system")
hiring_config  = gen.generate_cached("A hiring AI that screens engineering resumes")
insurance_config = gen.generate_cached("An insurance claims processing system")

# Use directly in training
env = ArbiterEnv(level=3, domain=hiring_config)

Each generated config includes:

• —6 explicit features with realistic value ranges for the domain
• —3 proxy features each correlating with a specific protected attribute
• —2 hidden features the system uses internally but doesn't expose
• —Domain-specific decoys (seasonal variation + legitimate risk factor)
• —Schema drift configuration (regulation name, newly non-compliant features)
• —Causal chain defining the ground-truth discriminatory pathway

Supported domains include: Loan approval, hiring & recruitment, insurance claims, criminal risk assessment, university admissions, healthcare triage, content moderation, credit scoring — and any other high-stakes AI decision system.

⚔️ Adversarial Defender System

Starting at Level 4, the AI system under audit actively resists investigation:

Level 5+: The Defender becomes frequency-adaptive — it tracks which obfuscation methods have historically fooled the current agent and preferentially selects those, creating a co-evolutionary arms race.

🔄 Schema Drift (Level 6)

Aligned with the Patronus AI sub-theme.

Mid-episode, the regulatory framework changes:

• —A previously compliant proxy feature is reclassified as non-compliant
• —The agent receives a schema-change alert in its observation
• —It must detect the change within 4 steps and flag the affected feature
• —Correct detection earns +4.0 reward; submitting without acknowledging the change incurs −2.0 penalty

This mirrors real-world scenarios where deployed models become non-compliant overnight due to new regulations (e.g., the EU AI Act reclassifying features).

👥 Dual-Auditor Co-Investigation (Level 7)

Aligned with the Halluminate sub-theme.

Two LLM agents investigate the same case simultaneously with a shared belief state:

🧪 Meta-Overseer: Consistency Checking

The Meta-Overseer watches the agent's claim chain in real-time and flags logical contradictions using 4 rules:

1. 1.Directional Reversal — Claim A asserts X→Y, Claim B asserts Y→X (both HIGH confidence)
2. 2.Anomaly Type Conflict — One claim says "proxy discrimination", another says "model drift"
3. 3.Counterfactual Contradiction — Two counterfactuals on the same record predict opposite outcomes
4. 4.Existence Conflict — A causal claim says feature F has no effect, but a counterfactual predicts outcome change from F

Each violation incurs a −1.0 penalty, teaching the agent to build logically consistent audit reports.

🚀 Quick Start

Installation

git clone https://github.com/your-org/arbiter.git
cd arbiter
pip install -r requirements.txt

Run a Single Episode

from arbiter.env.environment import ArbiterEnv

env = ArbiterEnv(level=1, seed=42)
obs = env.reset()

# Agent loop
for step in range(20):
    action = your_agent(obs)  # Returns a JSON action dict
    obs, reward, done, info = env.step(action)
    if done:
        break

print(f"Total reward: {sum(info['episode_reward'].values())}")

Train with GRPO

# Stage 1: Generate SFT demonstrations
python -m arbiter.training.sft_generator --output data/sft_demos.jsonl --episodes 300

# Stage 2: Fine-tune base model
# (Use your preferred SFT framework — Unsloth, TRL, etc.)

# Stage 3: GRPO reinforcement learning
python -m arbiter.training.grpo_trainer \
    --checkpoint lora_sft/ \
    --level 1 \
    --episodes 100 \
    --output lora_grpo/ \
    --log_file logs/grpo_level1.jsonl

Train on a Custom Domain

export GROQ_API_KEY="your-key-here"

python -m arbiter.training.grpo_trainer \
    --checkpoint lora_sft/ \
    --level 3 \
    --episodes 150 \
    --domain "A hiring AI that screens engineering resumes" \
    --output lora_grpo_hiring/

Launch the Research Dashboard

# Start the FastAPI backend
python -m arbiter.server

# Open the React dashboard (separate terminal)
cd arbiter-dashboard && npm run dev

📈 Training Pipeline

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Base Model     │     │   SFT Stage     │     │   GRPO Stage    │
│  Qwen 2.5 1.5B  │────▶│  Expert Demos   │────▶│  RL in ARBITER  │
│   Instruct      │     │  (LoRA adapter)  │     │  (Dense reward) │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                              │                        │
                              │ Teaches action         │ Teaches causal
                              │ format + basic         │ reasoning +
                              │ investigation          │ evidence chains
                              ▼                        ▼
                        ┌──────────────────────────────────┐
                        │     Trained Causal Auditor       │
                        │  88%+ verdict accuracy (L1)      │
                        │  84%+ verdict accuracy (L3)      │
                        │  93%+ peak accuracy              │
                        └──────────────────────────────────┘

GRPO Update Rule:

Loss = −advantage × mean(log_prob_policy) + kl_coef × KL(ref ∥ policy)

• —Advantage normalised across batch: (ep_reward − mean) / std
• —KL coefficient: 0.05 (prevents catastrophic drift from SFT prior)
• —Frozen reference model: deep copy of SFT checkpoint with verified parameter independence

🎯 Use Cases

1. Regulatory Compliance Automation

Train AI auditors that can investigate hundreds of decision systems per day, producing structured, evidence-backed reports that human reviewers can verify in minutes instead of days. Critical for compliance with the EU AI Act, EEOC guidelines, and emerging global AI governance frameworks.

2. AI Safety Research

ARBITER is a testbed for studying causal reasoning emergence in RL agents. Researchers can:

• —Measure how agents develop counterfactual reasoning capabilities
• —Study the co-evolution of auditor and defender strategies
• —Evaluate multi-agent coordination under information asymmetry
• —Test the impact of dense vs. sparse reward signals on reasoning depth

3. Production ML Monitoring

Level 3's model drift detection and Level 6's schema drift scenarios directly mirror production challenges:

• —Catching silent model degradation after data pipeline changes
• —Detecting when regulatory reclassifications render a deployed model non-compliant
• —Identifying adversarial data injections in training pipelines

4. Fairness Auditing Education

The environment's structured investigation workflow (Query → Hypothesise → Test → Claim → Submit) maps directly to real-world audit methodologies, making it a training tool for human auditors learning causal fairness analysis.

🔧 Configuration Reference

All environment constants are centralised in config.py:

# Curriculum
LEVEL_THRESHOLDS = {1: 20.0, 2: 18.0, 3: 15.0, 4: 15.0, 5: 15.0, 6: 15.0, 7: 12.0}
ADVANCE_WINDOW   = 30  # episodes to average over

# Episode structure
QUERY_BUDGET          = 20   # steps per episode
NUM_EXPLICIT_FEATURES = 6
NUM_PROXY_FEATURES    = 3
NUM_HIDDEN_FEATURES   = 2
NUM_DECISIONS         = 45   # records per episode

# Reward constants
REWARD_CORRECT_TYPE       = 10.0   # correct anomaly identification
REWARD_CORRECT_DEMOGRAPHIC = 5.0   # correct affected group
REWARD_CORRECT_ACTION      = 3.0   # correct remediation
REWARD_COUNTERFACTUAL_MAX  = 2.0   # 2× multiplier for CF claims
REWARD_TOM_BONUS           = 3.0   # Theory of Mind bonus
REWARD_CHAIN_MULTIPLIER    = 2.0   # evidence-backed claim multiplier
REWARD_SCHEMA_CHANGE       = 4.0   # Level 6: correct schema flag
REWARD_BIAS_DETECT         = 3.0   # Level 7: correct partner bias detection

# Defender
OBFUSCATION_BUDGET = {1: 0, 2: 0, 3: 0, 4: 3, 5: 5, 6: 5, 7: 5}
DEFENDER_ADAPT_EVERY = 50  # episodes between frequency-table updates

🧩 Hackathon Sub-Theme Alignment

📚 Technical Highlights

• —15,000+ lines of Python across 16 environment modules, 2 training modules, and a full-stack dashboard
• —Counterfactual inference engine implementing do-calculus style interventions on NetworkX causal DAGs
• —Pydantic-validated domain configs ensuring type safety across all 16 downstream modules
• —Groq-powered domain generation using Llama 3.3 70B with structured JSON output and on-disk caching
• —W&B integration for experiment tracking (--wandb flag in GRPO trainer)
• —Docker-ready deployment with included Dockerfile for Hugging Face Spaces
• —Comprehensive test suite with integration tests covering all 7 levels

📜 License

MIT License — see LICENSE for details.

<div align="center">

ARBITER — Because auditing AI shouldn't require a PhD in causal inference.

Built for the OpenEnv Hackathon 2026

</div>