stellahj/PRAGMA
🧩 PRAGMA PRAGMA is a benchmark for evaluating personalized guidance with memory alignment over lifelong conversation histories. It tests whether a model can use relevant past interactions while remaining aligned with a user's evolving experiences and trajectory. PRAGMA accompanies the paper “PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations” (EMNLP 2026). ✨ Dataset Overview Statistic Value Users 100 Queries 400… See the full description on the dataset page: https://huggingface.co/datasets/stellahj/PRAGMA.
🧩 PRAGMA
PRAGMA is a benchmark for evaluating personalized guidance with memory alignment over lifelong conversation histories. It tests whether a model can use relevant past interactions while remaining aligned with a user's evolving experiences and trajectory.
PRAGMA accompanies the paper [“PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations”](https://arxiv.org/abs/2609.09664) (EMNLP 2026).
✨ Dataset Overview
📂 Files
🧭 Query Types
🚀 Usage
Download the conversation histories, benchmark queries, and evaluation rubrics:
import json
from huggingface_hub import hf_hub_download
def load_pragma_file(filename):
path = hf_hub_download(
repo_id="stellahj/PRAGMA",
filename=f"data/{filename}",
repo_type="dataset",
)
with open(path, encoding="utf-8") as file:
return json.load(file)
full_sessions = load_pragma_file("full_sessions.json")
metadata = load_pragma_file("metadata.json")
response_metrics = load_pragma_file("response_metrics.json")Use full_sessions.json as the conversation history and metadata.json as the evaluation queries. Join them by user_id; for each query, evidence_session_index identifies the gold evidence sessions in the corresponding user's sessions array. Match the rubrics in response_metrics.json by (user_id, query).
alignment_metric evaluates whether the response is aligned with the user's relevant experience or trajectory. grounding_metric evaluates whether the response explicitly uses the relevant history. The experiments reported in the paper used GPT-5 (gpt-5-2025-08-07) as the default LLM judge.
🔗 Source and License
PRAGMA is constructed from sampled records from NVIDIA Privasis-Zero.
The dataset is distributed under the NVIDIA License and is intended for non-commercial research and evaluation use. See LICENSE for details.
🛠️ Construction Code
The benchmark construction pipeline and documentation are available in the PRAGMA GitHub repository.
📜 Citation
@misc{yu2026pragmaevaluatingpersonalizedguidance,
title={PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations},
author={Hyojeong Yu and Hyukhun Koh and Minsung Kim and Yunah Jang and Kyomin Jung},
year={2026},
eprint={2609.09664},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2609.09664},
}