Ace1213812/UnifiedMemBench-ParametricMemory

UnifiedMemBench-ParametricMemory

This repository contains the parametric-memory component of UnifiedMemBench, a benchmark suite for evaluating memory capabilities of large language models.

The parametric-memory component is derived from the same synthetic character timelines and long-dialogue construction pipeline used by UnifiedMemBench. It is designed to evaluate whether language models can internalize, update, arbitrate, and retrieve character-specific memories after training, rather than relying only on explicitly provided long-context input.

Dataset Configurations

This repository contains three configurations.

parametric_training_cpt

This configuration contains continual pre-training data for parametric memory. The data is derived from character timelines, memory records, long-dialogue slices, or memory-related text segments.

It is intended for next-token-prediction style training to inject character-specific memory into model parameters.

parametric_training_qa_sft

This configuration contains supervised fine-tuning data constructed from memory-oriented question-answer pairs.

It is intended to teach the model the expected answer format and improve task-oriented memory retrieval behavior after memory-related training.

parametric_training_test

This configuration contains held-out parametric-memory evaluation data.

It should be used only for evaluation and should not be included in CPT or QA-SFT training.

Recommended Usage

A typical experimental protocol is:

1. 1.Continue pre-train a model on parametric_training_cpt.
2. 2.Optionally apply supervised fine-tuning using parametric_training_qa_sft.
3. 3.Evaluate the trained model on the held-out parametric_training_test.

The parametric_training_test configuration should remain held out from all training stages.

Task Types

The parametric-memory component covers six memory-oriented task types:

• —Information Extraction (IE)
• —Knowledge Updating (KU)
• —Memory Arbitration (MA)
• —Temporal Reasoning (TR)
• —Multi-session Reasoning (MSR)
• —Event Summarization (ES)

These tasks are designed to evaluate different aspects of model memory after training, including factual recall, memory updating, conflict resolution, temporal reasoning, cross-session reasoning, and summarization of character events.

Relationship to Other UnifiedMemBench Components

UnifiedMemBench consists of three separately hosted dataset components:

1. 1.UnifiedMemBench-LongDialogue: long-dialogue memory benchmark data
2. 2.UnifiedMemBench-ParametricMemory: parametric-memory training and evaluation data
3. 3.UnifiedMemBench-StateTracking: dense state-tracking benchmark data

The parametric-memory component is derived from the same underlying synthetic character timelines and long-dialogue construction process used by the long-dialogue component. It contains training and evaluation data designed to test whether models can internalize, update, arbitrate, and retrieve character-specific memory after training. The state-tracking component is constructed from the same synthetic character event records, but evaluates memory through dense structured fact prompts rather than parametric training or natural dialogue.

Provenance

This dataset was generated through the UnifiedMemBench construction pipeline. The pipeline uses synthetic character profiles, event timelines, long-dialogue slices, memory-related text segmentation, QA construction, supervised fine-tuning data generation, held-out test construction, and automated quality-control procedures.

The construction process used MBTI-1024 Bank from the CharacterChat project as upstream seed character/profile data, with permission from the original authors. The upstream source is acknowledged in the accompanying paper, dataset documentation, and Croissant metadata.

License

This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license, following permission from the upstream data authors.

File Organization

The repository is organized as follows:

UnifiedMemBench-ParametricMemory/
  README.md
  parametric_training_cpt/
    stage_*.jsonl
  parametric_training_qa_sft/
    stage_*.jsonl
  parametric_training_test/
    *.jsonl