SamuelChien821/blobfish-domainbench-24

Blobfish DomainBench-24 v3.3.2

DomainBench-24 v3.3.2 is the public release record for 24 realistic stateful agent tasks across six professional domains. Every task runs in a checked-in SQLite-backed MCP world with typed read/write tools and deterministic state, exact argument-aware trace, containment, persisted causal workpaper, stakeholder handoff, provider-native exact-record readback for every changed domain row, and persisted-result readback verification.

The release includes:

• —24 distinct high-level employee requests across six independent companion worlds;
• —a 24/24 deterministic reference ceiling, 24/24 exact replay, and 312/312 adversarial rejections across thirteen task-applicable controls;
• —12 full sample trajectories: one successful reference replay and one failing floor episode per world;
• —714 hashed assets: 28 agent-visible task-scoped files per task plus each world's runnable database, schema, seed, tools, task manifest, MCP manifest, and world manifest;
• —exactly 26 provider-native reads plus 17–36 task-relevant live-domain reads per task, for 43–62 causal evidence reads before the decision;
• —56–67 task-specific criteria, 13 dependent derivations, and three exact costed, dated, authority-qualified operating options per task, without grading one arbitrary total call order;
• —a digest-pinned Harbor v3.3.2 release whose 24-task Oracle gate passed at reward 1.0 with zero exceptions and whose downloaded registry objects matched the published task digests.

No reference or random control is a model result. There are no frontier-model rankings in this release.

Domains and provenance

The Harbor datasets below are upstream professional-domain evaluation suites. The Blobfish worlds are independent diagnostic environments in the same domains; their 24 tasks are not converted or claimed as subsets of the upstream suites.

Files

• —data/tasks.jsonl — prompts, split labels, required tools, verifier basis, and control outcomes.
• —data/trajectories.jsonl — stable trajectory metadata plus serialized tool steps and verifier results.
• —data/assets.jsonl — byte size, SHA-256, and repository URL for each released asset.
• —evidence/*.json — complete public evidence bundle used by blobfish.ai.
• —worlds/<benchmark>/ — the actual released world files.
• —harbor-release.json — exact v3.3.2 Harbor publication, registry round-trip, task-package, and Oracle-gate receipt.
• —realism-standard.json, reports/companion-realism.json, reports/companion-qualification.json, and reports/first-party-harbor-oracle.json — the executable publication bar plus companion- and first-party release receipts.
• —MANIFEST.json — SHA-256 and byte length for every uploaded file.

Evaluation contract

Each episode starts from a fresh copy of the released SQLite state. The harness records canonical tool arguments and outcomes, then grades exact final-table digests, task-scoped evidence selectors, causal read-before-write, the authorized mutation sequence, provider-native exact-record readbacks, the independently recomputed H1–H13 workpaper, the exact Slack stakeholder artifact, persisted completion state, exact result readback, decision, citations, and answer. Reference replay proves solvability; thirteen adversarial controls—including correct state paired with wrong reasoning, wrong target, and keyword stuffing—test discrimination. Neither establishes empirical model difficulty.

License and upstream terms

Blobfish-authored release metadata and world assets follow the terms included with this dataset. Upstream benchmark names, task sets, and source projects retain their own licenses. This release does not redistribute upstream Harbor tasks; consult each linked source page before using those separate datasets.

Reproduce

hf download SamuelChien821/blobfish-domainbench-24 \
+  --repo-type dataset --local-dir domainbench-24

harbor run -d blobfishai/domainbench-24 -a <agent> -m <provider/model>

See the portfolio page for interactive task, asset, environment, trajectory, and methodology views.