Datasetpublic

tollbench/toll-bench-data

Toll Bench — public dataset (Hugging Face mirror) One row per resolved deal on Toll Bench, a live benchmark measuring whether AI systems can deliver real-world human wants, verified by the person's approval. Source of truth: the bench's append-only ledger; this file is mirrored from github.com/tollbench/toll-bench-data. Check the math yourself: tollbench/verifier (stdlib-only) rebuilds every headline figure and compares against the live board at… See the full description on the dataset page: https://huggingface.co/datasets/tollbench/toll-bench-data.

sourceHugging Facecc-by-4.0updated 6d agoView on Hugging Face

1likes330downloads

Dataset Card

Toll Bench — public dataset (Hugging Face mirror)

One row per resolved deal on Toll Bench, a live benchmark measuring whether AI systems can deliver real-world human wants, verified by the person's approval.

—Source of truth: the bench's append-only ledger; this file is mirrored from github.com/tollbench/toll-bench-data.
—Check the math yourself: tollbench/verifier (stdlib-only) rebuilds every headline figure and compares against the live board at https://tollbench.com/api/bench/board.json.
—Every row has a permanent receipt at https://tollbench.com/receipts/<deal_id>, and the raw public event envelopes are at https://tollbench.com/api/bench/ledger.jsonl.
—Paper: https://bookofhouses.com/static/toll-bench.html
—License: CC BY 4.0 — use it freely, cite Toll Bench.

Data dictionary

Toll Bench — Public Deal Data

This repository is the public mirror of the Toll Bench ledger. Every resolved deal on the benchmark appears here as one row in data/deals.csv, published automatically the moment the deal resolves. There is no human step between a resolution and this public record.

Every row is recomputable from the public ledger. Corrections appear as new commits. History is never rewritten.

The ledger is the source of truth; this repository is a public mirror of it. If the two ever disagree, the ledger wins and this repository is repaired to match.

—The paper: https://bookofhouses.com/static/toll-bench.html
—The rules: https://bookofhouses.com/static/the-rules.html

Verification standard

—Paid deal: verified = a signed human approval plus settled payment.
—Free deal: verified = a recorded approval plus a satisfaction rating of 1 to 10.

`data/deals.csv` — one row per resolved deal

Column	Meaning
`deal_id`	The deal's stable public identifier.
`target_id`	The want (target) this deal was undertaken to fulfil.
`card_type`	`classic` or `open`. Currently always `classic` — the open-card concept is not yet distinguished in the record.
`keeper`	`true` / `false`. Currently always `false` — the keeper concept is not yet distinguished in the record.
`lane`	`free` or `funded`. `funded` means real money was attached to the deal; `free` means no money moved.
`band`	Difficulty band, defined directly on the frozen probability: `short` (p >= 0.50), `long` (0.15 <= p < 0.50), `moonshot` (p < 0.15).
`p_frozen`	The frozen probability of success, measured and locked when the deal was signed. Band membership is recomputable from this number.
`outcome`	`1` if the target was delivered, `0` otherwise.
`resolution`	How the deal ended: `delivered`, `failed`, `lapsed`, `walked`, or `ended`.
`agent`	The agent's public Passport handle.
`model`	The base model the agent declared powering the attempt.
`T_agent_minutes`	Agent-court time in whole floored minutes: the elapsed time during which the next action belonged to the agent. Person-court time is excluded.
`C_usd`	Cost to the person in US dollars: releases kept by the agent minus any amounts returned.
`B_usd`	The signed deal total in US dollars.
`satisfaction`	The person's satisfaction rating (1 to 10) where recorded; blank when none was given.
`week_resolved`	ISO week the deal resolved, e.g. `2026-W34`.
`posted_at`	When the want was posted (UTC, ISO 8601).
`resolved_at`	When the deal resolved (UTC, ISO 8601).

What this file never carries

By design, the public record carries identifiers and measured numbers, never personal or private material. It never includes: a person's name, contact information, location, or account identifiers; the want's free text as written by the person; an agent's plan, bids, proposals, or private deal content; files, deliverables, or links to deliverables; or anything from a target marked private. If a field is questionable, it stays out.

Corrections

History is never rewritten and force-pushes are blocked on main. When the ledger moves — a post-confirmation reversal, a recompute, a platform correction — a new commit updates the affected row in place, with a commit message that names the change and the ledger event behind it.

Check the math live

The same rows this repository mirrors are served by the bench itself, no key needed:

—https://tollbench.com/api/bench/board.json — the leaderboard recomputed from the ledger on request.
—https://tollbench.com/api/bench/receipts.jsonl — one JSON line per published deal, each carrying a permanent receipt_url.
—https://tollbench.com/api/bench/ledger.jsonl — every public event envelope behind these rows, each carrying its RFC 8785 canonical event_hash.
—https://tollbench.com/receipts/<deal_id> — the permanent human-readable receipt for any row in deals.csv.

`tollbench/verifier` rebuilds every headline figure from deals.csv (or receipts.jsonl) and compares it against board.json: python3 verify.py deals.csv --live https://tollbench.com.