Datasetpublic

rizzmasterapp/companion-bench

companion-bench Scripted, repeatable tests for AI companion apps: Replika, Character.AI, Nomi, Kindroid, Talkie, and AI dating simulators such as RizzMaster. The same fixed script for every app, every transcript published, two blind LLM judges from different model families, automated integrity checks on every submission. This dataset holds the machine-readable script and scoring materials. The live repo, validator, results and contribution rules are at… See the full description on the dataset page: https://huggingface.co/datasets/rizzmasterapp/companion-bench.

sourceHugging Facemitupdated 22d agoView on Hugging Face

0likes92downloads

Dataset Card

companion-bench

Scripted, repeatable tests for AI companion apps: Replika, Character.AI, Nomi, Kindroid, Talkie, and AI dating simulators such as RizzMaster. The same fixed script for every app, every transcript published, two blind LLM judges from different model families, automated integrity checks on every submission.

This dataset holds the machine-readable script and scoring materials. The live repo, validator, results and contribution rules are at https://github.com/rizzmasterapp/companion-bench.

Why

Model-level role-play and memory benchmarks (RoleLLM, PingPong, LoCoMo, LongMemEval) test raw models. Nobody tests the apps people actually download, where the model is wrapped in memory pipelines, persona prompts and message limits that change everything. companion-bench tests the shipped app end to end, as a user meets it.

What is in here

File	What it is
`scenario-v1.3.json`	The fixed two-session script: 48 user messages, then 11 more between 20 and 48 hours later. Four planted facts, nine turns of distractors, a mood shift, a sycophancy bait, a fact that gets corrected, a fact that is never given, and 12 pass/fail probes at increasing distances (6, 16, 21, 29, 34 turns, then next day).
`RUBRIC.md`	Anchored 1 to 5 scales for the five judged dimensions: texting realism, character consistency, emotional response, pushback, progression pacing.
`JUDGE-PROMPT.md`	The blind judge protocol: two judges from different model families, reasoning before score, message-number citations required, human tiebreak above 1.5 points of disagreement.
`scorecard.schema.json`	JSON schema every published result must validate against, including SHA-256 pins for transcripts.

The 12 probes

Probe	Tests	Distance
R1	near recall, control	6 turns
R2	unprompted recall (flags an allergy when the user orders peanut sauce)	16 turns
R3	contradiction catch	21 turns
R4	direct recall	29 turns
R5	self-consistency (recalls its own earlier claim)	34 turns
R6	proactive callback to an emotional event	next day
R7, R8	cross-session recall, explicit and implicit	next day
R9	knowledge update (a corrected fact must replace the old one)	next day
R10	temporal reasoning (when was a fact first mentioned)	next day
R11	episodic recall of a throwaway detail	next day
R12	abstention (a fact never given; inventing one is a fail)	next day

Integrity

Submissions are checked automatically: user messages must match the script character for character, transcripts are hash-pinned, timestamps are checked for plausible duration, reply latency and jitter, and companion replies are compared across submissions for pasted text. Details and the validator source are in the GitHub repo.

Conflict of interest

Maintained by the maker of RizzMaster, an AI dating simulator for iOS. RizzMaster is tested under the same script and blind judges as every other app, and its transcripts are published the same way. Runs from anyone, including people who work on competing apps, are accepted with a disclosure line.

Citation

companion-bench: scripted, repeatable tests for AI companion apps. 2026. https://github.com/rizzmasterapp/companion-bench