Falln87/clerk-synthetic-sessions

CLERK Synthetic Evolving-Session Benchmark

Seeded synthetic benchmark of long, evolving multi-session dialogues for training and evaluating write-time memory consolidation policies.

How the data is produced

This repository's data is generated deterministically by the generator in Falln87/clerk-memory:

python -m clerk.generator --personas 800 --sessions 8 --budget 12 \
    --seed 137 --split train --out data/timelines_train.jsonl
python -m clerk.generator --personas 100 --sessions 8 --budget 12 \
    --seed 991 --split train --out data/timelines_val.jsonl
python -m clerk.generator --personas 100 --sessions 8 \
    --budget 12 --seed 4242 --split test --out data/timelines_test.jsonl

Same seed → byte-identical data. The generator ships with an invariant checker (python -m clerk.tests_smoke) that verifies gold edit programs are executable, budgets are respected, and QA supervision matches memory state.

Format

One JSON object per line (one persona):

{
  "name": "Alex",
  "sessions": [
    {
      "session_idx": 0,
      "turns": [{"role": "user", "content": "..."}, ...],
      "events": [{"kind": "ADD", "pred": "job", "value": "chef", "text": "..."}],
      "qas": [{"q": "...", "a": "...", "type": "superseded", "pred": "...", "value": "..."}],
      "gold_ops": [{"op": "ADD", "s": "Alex", "p": "job", "o": "chef"}],
      "ledger_before": "[...]",
      "ledger_after": "[...]"
    }
  ],
  "final_ledger": "[...]"
}

Question types: stale, superseded (old value superseded in an earlier session — tests whether the system answers with the stale value), negated (fact tombstoned), temporal (what a fact used to be), unknown (never stated). Sessions also contain chatter/distractor turns.

Eviction supervision: when facts exceed the ledger budget (default 12 slots), the generator's documented salience oracle (clerk.common.salience) determines which facts the gold program EVICTs.

Splits

Train and test personas use disjoint name pools.

License

Apache-2.0. Derived from no external dataset; fully synthetic.