TonicAI/knowledge-worker-search-bench

Knowledge-Worker Search Bench

40 multi-channel retrieval tasks over realistic synthetic knowledge-worker environments, generated with Tonic Fabricate.

Each task drops an agent into one persona's work world — mail (Outlook or Gmail), Slack, Google Docs, calendar, attachments — and asks a question a real chief-of-staff-style assistant would get: "brief me for tomorrow's sync", "where did we land on the renewal, and what forced the timeline?". Answering requires finding and synthesizing evidence scattered across channels; grading is per-claim (an LLM judge checks each rubric claim is conveyed and grounded in cited artifacts).

Code, docs, and eval harness: github.com/TonicAI/knowledgeworkersearch_bench (a verifiers environment). This dataset stores the tasks and the environment data; the GitHub repo is the central source.

Contents

Row schema (tasks.jsonl)

• —task_id, persona, shape (multi_hop_chain 32 / thread_synthesis 6 / attachment_lookup 2), difficulty (easy 20 / medium 18 / hard 2, banded by mean reward across a six-model panel), today (the persona's current date)
• —system — persona framing; prompt — the user request + output contract
• —rubric_claims — what a correct answer must convey; evidence_artifact_ids — where the ground truth lives (resolvable in the persona DB); evidence_channels — channel spread of the evidence
• —reference_rewards — per-task rewards from the published six-model panel (unified per-claim grader, single rollout)

Reference results

Every task is verified solvable (≥1 frontier model scores ≥0.8), yet the slate separates models cleanly.

The environments

Persona DBs are fully synthetic worlds: consistent orgs, characters, storylines, and timelines woven across outlook__* / gmail__* / slack__* / google_docs__* / *calendar__* tables. No real people, companies, or data. The eval harness exposes them through read-only product-shaped tools (search/read for each channel + attachment parsers); agents never query SQL directly.

Usage

# Browse tasks
from datasets import load_dataset
ds = load_dataset("TonicAI/knowledge-worker-search-bench", split="test")

# Run the eval (see the GitHub repo)
pip install knowledge-worker-search-bench  # or: pip install -e . from the repo
python scripts/run_eval.py --models claude-haiku-4-5

The harness resolves persona DBs and task specs from this dataset automatically via huggingface_hub when they aren't bundled locally.