yonsei-dli/Persona2Web

Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

Paper | Project Page | GitHub Repository

Persona2Web is a benchmark for evaluating personalized web agents on the real open web.

Dataset Structure

Data/
├── README.md
├── data/
│   ├── ground_truth.jsonl
│   ├── query_personalization_0.jsonl
│   ├── query_personalization_1.jsonl
│   ├── query_personalization_2.jsonl
│   └── user_history.jsonl
├── ground_truth/
│   └── *.json  150 files
├── query/
│   ├── personalization_0/
│   │   └── *.json  150 files
│   ├── personalization_1/
│   │   └── *.json  150 files
│   └── personalization_2/
│       └── *.json  150 files
└── user_history/
    └── *.json  50 files

Data Components

data/

The data/ directory provides JSONL versions of the main dataset components for convenient loading and processing.

• —ground_truth.jsonl contains the ground-truth personalization information for all tasks.
• —query_personalization_0.jsonl contains Level 0 queries, where both website and preference constraints are explicit.
• —query_personalization_1.jsonl contains Level 1 queries, where preference constraints are explicit while website information is masked.
• —query_personalization_2.jsonl contains Level 2 queries, where both website and preference constraints are masked.
• —user_history.jsonl contains user histories that reveal preferences implicitly through behavioral patterns.

ground_truth/

The ground_truth/ directory contains 150 individual JSON files, one for each Persona2Web task. Each file provides the reference information used to evaluate whether the agent correctly resolves the personalized query.

query/

The query/ directory contains task queries organized by ambiguity level.

• —personalization_0/ contains 150 clear queries with explicit website and preference constraints.
• —personalization_1/ contains 150 queries where the website is masked and must be inferred from user history.
• —personalization_2/ contains 150 fully ambiguous queries where both website and preference constraints must be inferred from user history.

user_history/

The user_history/ directory contains 50 JSON files of browsing-style user histories. These histories provide contextual user behavior over long time spans, where preferences are embedded implicitly rather than stated as explicit profile descriptions.