roychowdhuryresearch/Panini-Benchmarks

Panini: Continual Learning in Token Space via Structured Memory

Extracted Generative Semantic Workspace (GSW) representations and curated evaluation splits from [Panini](https://arxiv.org/abs/2602.15156), provided for ease of replication and future research.

Contents

GSW Networks (gsw_networks/)

Structured semantic representations extracted from document corpora using the Panini/GSW framework. Each GSW captures entities, their roles/states, and verb-phrase relationships as question-answer pairs.

Format: Each line is a JSON object with:

{
  "doc_id": "doc_0",
  "source_file": "gsw_0_0.json",
  "gsw": {
    "entity_nodes": [...],
    "verb_phrase_nodes": [...],
    "space_nodes": [...],
    "time_nodes": [...],
    "similarity_edges": [...],
    "space_edges": [...],
    "time_edges": [...]
  }
}

Corpus source: All evaluation corpora follow the HippoRAG v2 splits. GSWs were generated using GPT-4o with the Panini operator (see paper for details).

Paper Results

Main Results (GPT-4o-mini)

Platinum Evaluation Splits (platinum/)

To evaluate reliability under missing evidence, we curate Platinum splits for MuSiQue and 2WikiMultihopQA. Each question is verified for answerability through multi-model consensus and labeled as answerable (gold answer is derivable from supporting documents) or unanswerable (evidence is missing or ambiguous). This allows measuring both QA accuracy and abstention quality. See Section 4.2 of the paper for full construction details.

Files:

• —2wiki_platinum.json — Answerable 2Wiki questions
• —2wiki_unanswerable.json — Unanswerable 2Wiki questions
• —musique_platinum.json — Answerable MuSiQue questions
• —musique_unanswerable.json — Unanswerable MuSiQue questions

Platinum Results (GPT-4o-mini)

Usage

import json

# Load GSW networks for a dataset
gsws = []
with open("gsw_networks/2wiki.jsonl") as f:
    for line in f:
        gsws.append(json.loads(line))

# Load platinum evaluation split
with open("platinum/2wiki_platinum.json") as f:
    platinum_questions = json.load(f)

With the Panini/GSW-Memory package

from gsw_memory.qa.gsw_tools import GSWTools

# Write GSWs to individual files for GSWTools
import os, json
os.makedirs("gsw_files", exist_ok=True)
with open("gsw_networks/2wiki.jsonl") as f:
    for line in f:
        record = json.loads(line)
        doc_dir = f"gsw_files/{record['doc_id']}"
        os.makedirs(doc_dir, exist_ok=True)
        with open(f"{doc_dir}/{record['source_file']}", "w") as out:
            json.dump(record["gsw"], out)

# Use GSWTools for retrieval
import glob
gsw_files = glob.glob("gsw_files/doc_*/*.json")
tools = GSWTools(gsw_files)
tools.build_index()
results = tools.search_gsw_bm25("Lothair II", limit=5)

Attribution

• —Evaluation corpora follow splits from HippoRAG v2 (Gutiérrez et al., 2025)
• —2WikiMultihopQA from Ho et al., 2020
• —MuSiQue from Trivedi et al., 2022
• —HotpotQA from Yang et al., 2018
• —Natural Questions from Kwiatkowski et al., 2019
• —PopQA from Mallen et al., 2023
• —Platinum curation methodology inspired by Vendrow et al., 2025

Citation

@misc{rajesh2026paninicontinuallearningtoken,
      title={Panini: Continual Learning in Token Space via Structured Memory},
      author={Shreyas Rajesh and Pavan Holur and Mehmet Yigit Turali and Chenda Duan and Vwani Roychowdhury},
      year={2026},
      eprint={2602.15156},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.15156},
}

License

MIT