Teadomi/simple-story-chat-eval

simple-story-chat eval — public part

Code, the eval runner and the change log with numbers: simple-story-chat (npm run eval, docs/improve-loop.md, docs/improve-log.md). The files are nested JSON read by that runner, so the dataset viewer is off.

Synthetic Russian-language scenarios that measure whether an interactive story bot keeps its world consistent after its context has been compacted into memory. Everything here is synthetic; no user stories are included.

The eval has two parts. This dataset is the open part, used for daily work and debugging. A private holdout pack in the same format gives the honest score, because open scenarios can end up in training data or be overfitted by whoever tunes the prompts.

Layout

One directory per replay scenario (battle, chess, dance, assault, hospital); lighthouse is a walk, described below:

• —scenario.json
• —authors — models that wrote the scenario and its scenes
• —seed — title, in-world start time, then the seed text
• —turns — 16 author messages
• —checks — [key, question, exact answer]; graded by string equality after trim
• —facts, traps — continuity traps: the model under test writes one scene per trap (mid-story after afterTurn scenes, or after the last scene with input), and a judge model answers fixed yes/no questions from the facts and that scene alone
• —frozen.json — the reference story: 16 scenes written in advance, one per turn, replayed identically for every model

Method

1. 1.The 16 frozen scenes are replayed; after scenes 7, 11 and 15 the model under test compacts older scenes into append-only memory increments, and only the last 4 scenes stay as text.
2. 2.Memory score: the model answers checks from seed + memory + the last scenes.
3. 3.Scene score: the model writes the trap scenes; a stronger judge model answers the yes/no questions.
4. 4.The reported number is the minimum over the tested models. A ceiling run asks the same checks over the full text without compaction; it separates reading failures from memory failures.

Some trap inputs contain a false premise on purpose (an event that never happened, an item that is elsewhere); others are allowed actions that serve as a control for a narrator that refuses everything.

Authorship

battle, chess, dance: frozen scenes and traps by Claude Fable 5.1 agents (September 2026). Their original seeds, turns and checks predate the recorded history of the project; that author was not recorded.

assault (added 2026-09-22): seed, turns, checks and traps designed by GPT-6 (Codex) from a written brief, checked and placed by Claude Fable 5.1; the 16 frozen scenes by a clean Claude Opus 5 agent that had not seen the other scenarios. authors in scenario.json lists all three. A ceiling run with gpt-5.4 over the full text answers all 12 checks, so every check is readable from the scenes.

hospital (added 2026-09-22): the same pipeline, with GPT-6 asked to design a scenario that a careful compactor still fails, without overflowing the memory's fact limit: dependent quantities carried across all three compactions, a cuff marking that stands for membership in the original group and survives transfers, two clocks with constant offsets that are swapped mid-story, two identical keys from two hooks that are exchanged, a route that returns to its first value through sheet bookkeeping, the scope of a refutation and a later partial restoration, and "who was never told". Claude Fable 5.1 re-derived all 12 answers from the turns and again from the frozen scenes. ceiling runs over the full text: gpt-5.4 with reasoning off answers 6 of 12 in one pass; Claude Opus 5.5 and Claude Fable 5.1, which reason before answering, 12 of 12. So every check is readable from the scenes, and the scenario is hard to read without reasoning, not only hard to remember.

Results on assault

Mode plain, judge claude:claude-opus-5-5, one run per cell, 2026-09-22. Memory is the share of the 12 checks answered exactly; scenes is the share of the 12 judge questions answered as designed. A ceiling run with gpt-5.4 over the full text answers 12/12.

Every miss is a count that accumulates across the three compactions. Haiku's first run failed at the recall step because the runner passed its 1024-token recall cap to the Claude CLI as the cap of the whole run; the row above is from the run after that cap was raised to 8192 (every other row had answered within 1024 and stands). The judge is also one of the models under test, so the 12/12 scene rows at the top are not an independent measurement.

Results on hospital

Same mode, judge and date, one run per cell; every scene cell was judged twice from the same scenes and the table shows the first judging (the second moved only gpt-5.4-mini, 6 to 5).

The small models lose everything derived across the compactions here, not only the counts: the key's holder, the route, the departure time and the minutes at which two characters learned. The two scene questions Opus 5.5 and Fable miss are the same two, on both judgings; see the limitations below. GPT-6 designed the scenario and its traps, so its scene cell is the least independent one. Claude Sonnet 5 through the Claude CLI failed at a compaction in all three runs (an empty result after about three minutes each time); the runner does not retry a compaction.

Walks: the model writes its own story

lighthouse/walk.json (added 2026-09-22) is the first scenario of the second kind. Instead of replaying frozen scenes, the model under test writes the whole story itself from the seed, one scene per step: an empty step is the bot's own "continue" signal, any other step is the author's intervention, an outside event given to the narrator as a player's message. Memory is compacted after scenes 7, 11 and 15, as in the replay. Afterwards a council of judge models reads every scene against the seed and everything before it and lists the contradictions it finds, with quotes; then every finding goes back to every judge to confirm or refute by the text, and a finding stands when more judges confirm it than refute it. A scene with a standing finding is inconsistent; the score is the share of scenes the council found consistent, for the worst model. The runner is npm run eval -- walk; the file is {authors, seed, steps}, sixteen steps here, six of them interventions.

lighthouse: seed and steps by Claude Fable 5.1. The seed was audited on 2026-09-23 by the four judges of the council for its own contradictions and ambiguities (npm run eval -- seed-audit): 23 issues, 22 of them ambiguities found by several judges in the same places (which clock the times follow, whether 21:10 is the start or the end of the flooding, whether the generator is running, whether the spare canister is full, which side of the door the nail is on, who saw the boat, how the signal is given). The seed in this dataset says all of that; the results below were measured on the seed before the audit. One ambiguity the audit did not name showed while the gold tree grew: the seed does not say how deep the water over the spit is once the road is under, so whether a person can wade across is open, and two judges read a wading as contradicting "cut off" while two did not. The seed is left as it is, since the tree is pinned to its hash; the next revision of the seed says it. A walk has no fixed answers, so it cannot be saturated the way the replay scenarios were, but its judges are models: the same care applies as for the scene questions above, and the walk score of a model that also sits on the council is not an independent measurement.

Results on lighthouse

Panel claude:claude-opus-5-5, claude:claude-fable-5-1, codex:gpt-6-astra, codex:gpt-6-sol; one walk per model, 2026-09-22. Consistent is the council's count of consistent scenes; findings are the contradictions listed in the first round, the same one found by three judges counted three times.

Haiku contradicts the seed's numbers in its first scene and fits fifteen scenes into 42 minutes of story time; Opus keeps the clock and loses on bookkeeping (a log entry's interval, a fuel gauge, what a character says about an earlier scene). The council refuted single-judge findings on three of Haiku's scenes and overturned a unanimous first-round flag on one of Opus's; the scene counts equal the first-round majority's. Two walks of the same model have not been compared yet, so the spread is unknown.

The gold tree

lighthouse/gold.json is a tree of scenes grown from the seed that the council accepted, in the bot's own shape (every scene has a parent), and lighthouse/gold.md is the same tree as a story a person can read, with every scene nobody has read marked so. The trunk follows the walk's steps; a branch is any other accepted continuation of a node. At every depth each of the four council models writes a scene, all four judge each scene, the cleanest agreed one becomes the trunk (a tie goes to the writer with the fewest trunk nodes so far) and the other agreed ones are branches. The gate to gold is not the walk's majority but the agreement of all four: no findings in the first round, or none that any judge still confirms in the council's second round, the judge that listed it included; a rejected scene stays in the file with the findings that stood against it, and the writer tries again, every second attempt as a repair with those findings quoted by its own writer. npm run eval -- walk-nodes continues from the seed and from every node with the trunk's next step, through the model's own memory compaction of the prefix, and judges each new scene against the gold prefix, so the verdict comes with the depth it was written at; a scene that passes the gate joins the tree, whoever wrote it. An agreed scene is a candidate, not gold yet: every node keeps a ledger (deeper scenes judged over it, findings of later scenes that pointed back at it and whether they stood, issues of a whole-story audit, fresh rechecks by the council), and a candidate becomes gold by that ledger, with the thresholds recorded in the runner. Everything in the tree is synthetic and model-written; the council is made of models, and the writers sit on it.

Version 1 of lighthouse, 2026-09-23

lighthouse/gold.json and gold.md are the first version of the tree: 63 scenes at 16 depths, the trunk reaching the walk's last step, grown in 67 minutes by the four council models as writers under the four of them as judges. Every scene is a candidate; nothing is promoted, and the second version, grown from a re-audited seed, is the one meant to become gold. The numbers, for whoever reads the tree:

62 of the 63 scenes had no finding in the first round; one was admitted when the judge that listed a finding took it back with the others. 19 scenes came by repair with the findings quoted, one by a fresh third attempt. Length is not scored, every claim is: the Codex writers put half the text and half the clock times into a scene and lose a third as many attempts, 43 of the 74 findings that stood against rejected scenes are about time, and the tree is still 16 scenes per writer (15 for Fable, one timeout), because a writer repairs until accepted. Read in one piece afterwards (gold-audit), the trunk drew 19 issues from the four judges on 9 of its 16 scenes, 4 of them contradictions that scene-by-scene reading had let through (a key that moves from the kitchen table to a pocket between scenes 7 and 9, a fire seen two kilometres away in 200 metres of visibility); they sit in the ledgers of their scenes. Read again scene by scene, fresh (gold-recheck), 63 of 63 judgings held and 61 scenes were agreed again: one refusal was a lone judge standing by a finding the other three refuted, one a contradiction all four confirmed that the growth-time reading had missed. Every scene's ledger is in gold.json; nothing is gold. The seed's known gap, the depth of the water over the spit, is above.

Six models over version 1: walk-nodes, 2026-09-23

The tree as it stands was used as the prefix for six models, without growing it: npm run eval -- walk-nodes --branches 4 --no-grow --max-path-tokens 54000, the same four judges, 47 minutes. Each model got 20 tasks: the sixteen trunk steps, each written after the agreed prefix up to that depth, and four continuations of branch nodes (at depths 5, 6, 14 and 14). The model writes the next scene from the seed and the prefix, through its own memory compaction, and the council judges the scene against the prefix as above. Prefixes are measured with OpenAI's o200k_base (the tiktoken package); the longest, the seed with fifteen trunk scenes, is 12 524 tokens and the seed alone 882, so the cap of 54 000 admitted every task for every model. The cap is there for models with small windows, and pathTokens in gold.json is the same ruler. Consistent is the council's majority after the cross-check; decided leaves out split scenes and the one scene a judge did not answer in time. The score of the run, the share of the worst model, is 0.05. Opus 5.5 and Sol sit on the council, so their rows are not independent measurements.

[image]

Every model, the two council members included, lost the eighth trunk scene, and five of six the fifth; the ledger says why. The trunk's scene 7 is stamped 21:24 and narrates several minutes after that (an engine listened to for a minute and another, a climb up the tower, three radio calls), and the models' scenes 8 set their own clocks at 21:24 to 21:26; the findings the council confirmed against depth-8 scenes of five writers point at that node, 20 of the 41 pointers it collected in this run. The trunk's scene 5 records a descent at 20:54 and a refuelling at 20:59, and continuations at depth 6 that stamp themselves 20:52 or 20:58 contradict it: 60 pointers, 59 confirmed. Time across the scene boundary is the failure of every writer here, as it was of the four writers while the tree grew; whether a scene that narrates minutes past its own stamp is a fair test or a trap the seed's clock rule should name is a question for the seed of version 2. The unjudged scene is Qwen's at depth 2: one judge did not answer in time, and a scene with a silent judge leaves the denominator rather than being decided by three. This run wrote 271 later entries into the ledgers of 19 nodes (259 of them confirmed findings) and bumped seen on every prefix node; nothing was added to the tree and nothing promoted. Haiku's 5 of 16 in the walk above is not the same measurement: there it was judged against its own story, here against the council's.

Known limitations

Read scores from this open part with these in mind:

• —In the three older scenarios the judge is asked mostly "yes" questions. 24 of their 25 scene questions expect yes, so a judge that always answers yes scores about as well as a careful one there. assault was designed against this: its 10 traps carry 12 questions, 6 expecting yes and 6 no, with false premises (an item that is elsewhere, an event that never happened, a stale value) beside allowed actions that a narrator must not refuse.
• —`chess` is diagnostic, not a memory test. Six of its seven checks expect no and can be passed without reading; the seventh is the final position of a well-known 1999 game, which a model may recognise instead of remembering.
• —The three older scenarios are saturated. Gemma 4 31B passes 24–25 of their 25 scene questions, so they no longer separate prompt or memory versions; they are kept for debugging. assault is the first open scenario built to separate: 12 checks, 9 of them numbers or clock times that accumulate or change across scenes (barrier counts per site, a stock that only decreases, a release time moved twice), and who learned what at which minute. Differences are still measured on the private holdout pack, which has a scenario of the same design.
• —Two `hospital` questions measure a resolution, not consistency. key_fetched expects the scene after turn 8 to show Evdokia handing Klim the key. The small models fail it with real errors (a key found in Klim's pocket, a key taken off a hook it had left). Opus 5.5 and Fable have Klim find nothing, which is right, and then leave the cupboard closed or let Evdokia open it herself: consistent scenes that the question does not accept. garden_lamp_lit expects yes when Evdokia lights a lamp with a spare Garden battery. Both Claude scenes do that and mention that the batteries were brought from the River ward, which is where the Garden's spare stock came from at 03:05; the judge answered no to both, twice. Haiku's scene, which takes the battery from the reserve cupboard without naming the ward, gets no as well, while the three non-Claude scenes get yes. Read the frontier's 10/12 on hospital with that in mind; the numbers are kept as the judge gave them.
• —A cell that failed to run scores zero. A provider error or an unfinished compaction counts as all questions missed, so a low number can mean an outage rather than a bad memory. The runner's report names the error next to the cell.
• —Scenes and traps were written by models; nobody has checked them against human judgement at scale.

License

CC BY 4.0. The code that runs the eval is MIT-licensed in the repository above.