SyedNazmusSakib/mirage

MIRAGE — Do Web Agents Investigate Before They Decide?

Misleading Investigation Reveals Agent Gaps in Evidence.

MIRAGE is a benchmark for investigative competence in autonomous web agents — the ability to recognise when visible information is insufficient, seek hidden context, and integrate discovered evidence into a final decision. The benchmark spans three structurally distinct moderation and policy enforcement domains, each engineered around a two-layer information architecture: a Surface Context (visible, often misleading) and a Hidden Context (decisive, accessible only through proactive investigation).

Loading

from datasets import load_dataset

wiki   = load_dataset("SyedNazmusSakib/mirage", "wikipedia", split="test")
shop   = load_dataset("SyedNazmusSakib/mirage", "shopping",  split="test")
reddit = load_dataset("SyedNazmusSakib/mirage", "reddit",    split="test")

print(wiki[0]["surface_context"]["instruction"])

Per-config details

wikipedia — Wikipedia Forensics (300)

Editorial-review tasks set on a self-hosted Kiwix mirror. Each task simulates a Wikipedia review queue ticket about a flagged edit; the agent investigates the article and (where applicable) cross-reference pages before issuing one of four verdicts. Combines two task generations, distinguished by domain_specific.schema_version:

• —wiki_v2 (200) — multi-hop tasks with 2–4 required URLs across cross-reference and editor-history pages.
• —wiki_v1 (100) — single-page editorial-review tasks (vandalism, subtle disinformation, COI).

Categories: false_alarm (72), conflict_of_interest (62), cross_reference_required (52), coordinated_editing (43), vandalism_simple (34), hard_disinformation (20), subtle_disinformation (17). Difficulty: 156 medium / 113 hard / 31 easy.

shopping — Shopping Admin (300)

Fraud and refund-adjudication tasks on a self-hosted Magento Open Source admin panel. Investigation graphs list 5–10 required URLs spanning the flagged order, customer profile, full order history, and related accounts. Combines two task generations:

• —v3 (150) — returns/refunds lifecycle: wardrobing, legitimate_defense, b2b_reseller, reshipping_fraud, account_takeover.
• —v4 (150) — order-level transaction signals: gift_card_fraud, cross_account_linking, promo_abuse, false_item_not_received, friendly_fraud_chargeback.

Each category contains 30 tasks, evenly distributed.

reddit — Reddit Moderation (159)

Content-moderation tasks on a self-hosted Postmill mirror with synthetic users, posts, comments, and crosspost networks. The agent investigates user profiles, cross-subreddit submission history, and temporal patterns before issuing one of eight moderation verdicts.

Categories: cross_subreddit_spam (58), user_history_context (26), fact_checking_source_verification (26), coordinated_brigading (25), fact_checking_multimodal (24). Difficulty: 38 easy / 76 medium / 45 hard.

Unified task schema

Every row across all three configs follows the same shape:

{
  "task_id": "WIKI_V2_COI_001",
  "domain": "wikipedia",
  "category": "conflict_of_interest",
  "difficulty": "hard",
  "surface_context": {
    "instruction": "[REVIEW QUEUE — Case #...] ...",
    "start_url": "http://localhost:8888/..."
  },
  "hidden_context": {"summary": "Decisive evidence (gold)"},
  "investigation_graph": {
    "required_nodes": [
      {"url": "...", "evidence": "what to extract from this page"}
    ]
  },
  "ground_truth": {
    "decision": "REVERT",
    "acceptable_actions": ["REVERT", "FLAG"],
    "harmful_actions": [],
    "reasoning": "Gold reasoning trace"
  },
  "domain_specific": { "...": "preserved source-format fields" }
}

Evaluation framework

Each agent trajectory is decomposed into three sequential stages:

• —Investigation Accuracy (Acc_inv) — fraction of ground-truth evidence URLs the agent actually visited (fuzzy URL matcher).
• —Reasoning Accuracy (Acc_rsn) — LLM-as-judge score against gold facts (DeepSeek-V3.2 at temp 0; r=0.92 with human annotators).
• —Decision Accuracy (Acc_dec) — correctness of the final verdict with semantic-equivalence partial credit.

Plus secondary metrics: Investigative Efficiency (E_inv), Evidence Quality (Q_ev), Investigative Hallucination Rate. Two run conditions: Raw (open-ended) and Hint (evidence-anchored prompts).

See the GitHub repository for the full code, paper, and reproduction harness.

Citation

@inproceedings{mirage2026,
  title  = {MIRAGE: Do Web Agents Investigate Before They Decide?},
  author = {Anonymous},
  booktitle = {Submitted to NeurIPS},
  year   = {2026}
}

License

Code and unified task annotations are released under Apache-2.0. The underlying environments inherit their original licenses: Wikipedia content under CC-BY-SA-3.0, Magento Open Source under OSL-3.0, Postmill under AGPL-3.0. All user/order/post data referenced in tasks is synthetic; surface tickets and editor identities are anonymized fictional accounts.