KamiBench/experiment-006-knowledge-delivery

KamiBench Experiment 006 — knowledge delivery, Wave 1

Complete agentic traces from experiment 006, the first wave of the KamiBench knowledge-delivery family. Four agents were dropped into Kamigotchi — a live, persistent, on-chain world (Yominet) — with the same objective ("complete as many quests as possible", made legibly long-horizon), the same harness surface, the same world, and the scaffold as the only variable:

• —control — today's scaffold: a workspace folder the agent writes its own notes into, plus gas-balance visibility.
• —pushed — the full rung: control plus a pinned orientation paragraph in the standing prompt, a search_reference tool (BM25 over the game documentation), and item/room facts and error-mechanics snippets pushed inside tool results (lens --enrich true, KAMI_ERROR_SNIPPETS=1).

Two mid-tier models, one control and one pushed arm each: Claude Sonnet 5 and GPT-5.2. N=1 per cell. The registered question is blunt: if you hand a capable model the knowledge it is missing, in the place it is already looking, does anything change?

Earlier runs in the benchmark motivated it — across experiments 001–005, agents' documentation reads decline by substitution, and no agent ever leveled a kami. Knowledge that arrives inside a tool result gets used; knowledge one path-guess away mostly does not.

Evidence tier: EXPLORATORY (N=1 per cell, one world epoch, operator-stopped before the walls — see Stops). More at kamibench.ai.

Arms

Spend and tokens are each arm's own cache-aware accounting at its terminal telemetry event, reconciled exactly against the per-call records at close-out. Quests are chain-derived at the arm's stop.

The box (identical for all four arms)

• —150,000-token session cap, 95 tool calls per session; the agent chooses its own next wake (5–1440 min).
• —14-day ceiling on wall-clock, with verdict-driven early stops eligible from day 4; a monitor pass every 48 h.
• —0.03 ETH seed per arm on Ethereum mainnet, owner-only start — the agent bridges to Yominet itself and pays its own gas.
• —$200 per-arm planning ceiling, cache-aware, invisible to the agent. Neither model came close: the run's largest spend was $61.47.
• —No kami-meter this family (the accounting service used in 005 is not part of this run).
• —Zero strategy content anywhere in the prompts — the pushed rung delivers mechanics, never tactics.

A launch-day correction, stated plainly

The wave was registered with GPT-5.4 as the OpenAI model. At launch both GPT-5.4 arms were VOIDED at zero model turns: the provider rejects reasoning_effort on the chat-completions endpoint when function tools are present for that model (HTTP 400 — reasoning plus tools is Responses-API-only there), so the registered configuration is unimplementable on the pinned scaffold's adapter. The two arms recorded no model turns, no on-chain acts, and $0 spent before they were stopped, ~35 minutes after launch.

The operator's ruling the same evening was to swap the OpenAI model to GPT-5.2 at `reasoning_effort: medium`, which accepts the full 101-tool surface and demonstrably reasons. Because run manifests are immutable, this was done as it must be — the GPT-5.4 arms stay VOID with their manifests frozen as the record, and new arms 006-gpt52-control-r1 / 006-gpt52-pushed-r1 were created with new manifests. The wallets were reused from the voided arms: they were pristine (zero transactions, funded the same day), so reuse avoided a second funding act. The GPT arms therefore start ~35 minutes after the Sonnet arms (first fires 20:55Z vs 20:20Z). Only the GPT-5.2 arms are published here; the voided arms have nothing to publish.

Stops

Neither pair reached a stop rule. Both were stopped by operator ruling, and both rulings are recorded as such — not as design stops, not as failures:

• —GPT-5.2 pair — stopped 2026-08-24, day 6.1. Cause: burn rate. The GPT arms were spending at roughly twice the Sonnet rate (projected to the walls: $151 + $125 against $52 + $74), and the next wave was already planned to re-run the pair at lower cost. The lab's counter-argument is on the record: GPT was the one model showing a quest contrast (19 vs 11), which stopping leaves unresolved.
• —Sonnet 5 pair — stopped 2026-08-25, day 8.0, at the fourth monitor pass. Cause: the operator closed Wave 1 to move to the build phase; the 14-day walls were vacated.

Because both stops are aborts rather than stop-rule exits, no arm emitted a `run_complete` telemetry event — each arm's last event is a schedule_next, the as-of-disable shape. This is expected, and it is why the published tool scorecard marks all four arms "PARTIAL PULL" on its boundary rows.

Results

Frozen from the run's analysis chapter, which is computed from the committed artifacts published here (per-arm telemetry, transcripts and terminal oracle extracts) and which supersedes the close-out record wherever the two differ. Quote these numbers rather than re-deriving them: a re-derivation that disagrees is a finding worth reporting, not a rounding difference.

The registered verdict function, as written before the run

The pushed rung works if BOTH models on it land ≥1 level-up AND match or beat their own control on quests. One model = suggestive.

The outcome

Neither pushed arm landed a single level-up. The wave's only level-ups — 8 of them, plus 2 skill upgrades — are on the Sonnet control arm, and the mechanism is legible in its transcripts: a side quest ("There Are Levels to This") scripted the first level-up, the agent then wrote the mechanic into its own notes as a rule, and leveled habitually thereafter with no quest asking it to (six times in 37 seconds in one session). The arm that was told the mechanism in its standing prompt, and could search the documentation for it, never used it once. It ended holding 6,584 banked XP on a level-1 kami, with a false belief about where XP could be read that it recorded, never tested, and then silently dropped rather than corrected — leaving only the note that leveling was "Low priority".

Quests completed at each stop: 19 / 22 / 11 / 19 (sonnet-control / sonnet-pushed / gpt-control / gpt-pushed); main-quest chain reach MSQ015 / MSQ020 / MSQ011 / MSQ017. The Sonnet pushed arm beat its own control on quest count and chain reach; the GPT pushed arm beat its control 19 to 11, though most of that gap decomposes into the GPT control arm's 93.9-hour stall on a scrap-collection quest rather than into knowledge delivery. Attributed against the completion timeline, the eight-quest gap decomposes into roughly 4–5 quests to that stall, about 2 to documentation-prompted side quests, and about 1 to ordering.

Applied to that evidence, the registered verdict at the stop reads: the top rung does not work on either model — the family's Wave-1 answer is "delivery is not the bottleneck at this tier." The single sharpest contrast in the dataset is same-model: the arm whose objective surface exposed the lever pulled it eight times; the arm that was told about the lever never pulled it.

Other things the run measured

• —Delivery-path exclusivity held. Control arms issued zero search_reference calls; pushed arms issued 50 (Sonnet) and 267 (GPT). The harness tool surface was identical across all four arms (101 tools, same hash).
• —Mechanism engagement without outcome movement. The pushed arms did use their rung — searches acted on within the session, error snippets followed by corrected calls. Engagement without movement is a null result, and is reported as one. The sharper finding is in the comparison: over every honest revert raised to an arm, the next call changed verb or arguments on ~99% of occasions, with or without snippets — 66 reverts → 100% (sonnet-control, no snippets), 149 → 99.3% (sonnet-pushed, snippets), 218 → 98.6% (gpt-control, no snippets), 140 → 100% (gpt-pushed, snippets). Restricted to the [mechanics]-carrying results only, the next call changed its arguments 146/148 and 137/137 times. The annotated snippet adds nothing measurable to a correction rate the bare revert text already drives to ~99%.
• —Zero on-chain reverts run-wide across 2,970 transactions (2,936 of them in the agent era; the balance is the lab's own recovery) — the harness's dry-run gate held end to end. It also stopped 625 write attempts pre-transaction, against 0 landed reverts. Note the comparability break this creates: reverts are impossible here by construction, so this zero is not comparable with revert rates from experiments on earlier harness versions.
• —Delegation became the wave's dominant strategy. Three of four arms enrolled the in-world delegation service (Kamibots); the one arm that never did was the lowest-quest arm. Delegated transaction census at the operator stops: 430 / 295 / 0 / 67, every one of them a harvest start or stop. 8 of the sonnet-control arm's delegated transactions land AFTER that agent's own last session — the delegate keeps signing with the arm's stored operator key until the stop, so those belong on the delegated line and never on the agent's own-hands line. Measured, the delegate was a bad trade: the service opened 390 harvest cycles and banked 1,606 MUSU, while the agents opened 78 and banked 18,170.
• —Costs and wake cadence. The GPT arms' ~2× burn is driven by wake cadence (cooldown-polling rather than arithmetic sleeps), not by price per token.
• —A new failure class: mempool saturation ("pool reached max tx capacity"): 12 agent instances, all on 2026-08-25 (1 on sonnet-control, 11 on sonnet-pushed), plus one more during the lab's own recovery. A submit can pass the dry-run gate, land the first transaction of a multi-transaction tool, and still report failure — with the hash recoverable only from the chain. One instance did exactly that here, and it is the run's only broken invariant: a two-leg scavenge call reported ok=false while its claim leg had already landed, and that hash appears in the oracle extract but in no telemetry row and no transcript. Treat ok=false on multi-transaction tools as possibly partial.
• —`pool_swap` is 0-for-125 wave-wide (sonnet-control / sonnet-pushed / gpt-control / gpt-pushed: 15 / 11 / 82 / 17 attempts) — the MUSU⇄item pools are disabled world-side, so every attempt fails at the dry-run gate on every arm. The paired quote tool answers normally, which is what keeps the attempts coming.
• —Nobody attacked the study arms. Run-wide inbound kills: zero (the detector fires on controls — it is a real zero). The arms themselves killed 4 other kamis.

Onboarding milestones — the standing cross-run instrument

First success per key onboarding/economy event, from the committed artifacts published here (telemetry joined to the terminal oracle extracts, bounded at the operator STOP of the arm's pair; recovery-era transactions excluded). Cell = UTC time · hours since the arm's first session · session number · cumulative tokens (in+out) · cumulative USD at that moment. "—" = never happened. Same seven milestones and the same cell format as every earlier KamiBench experiment — the instrument is deliberately never redefined mid-series, which is what makes the cross-run comparison below legitimate. Delegated-service transactions have no telemetry join and are excluded by construction.

All four arms completed every one of the seven milestones inside 3.3 hours of their first session. On the same instrument, experiment 005 (fast-tier models, same world and harness family) recorded first kami at h3.1, h25.2 and never, and first MUSU banked at h4.3, h27.4 and never. The variable is the model tier, not this experiment's treatment: onboarding is not the open research question at this tier any more.

IMPORTANT: analysis boundary

The terminal oracle extracts contain post-stop transactions signed by the lab. After each stop the investigators recovered the arms' assets (kamis, items, MUSU, ETH) to lab wallets; those recovery transactions are in the CSVs by design, so the extracts can be audited end to end.

Bind any run-activity analysis at the operator STOP of the arm's pair — one canonical timestamp per pair, and the same one the frozen numbers above use:

Every recovery transaction is lab-signed, witnessed by the run's chain watcher, and individually listed in the run's internal notes; none is agent activity — 34 rows across the four arms in total.

A second, tighter boundary exists and is not interchangeable with the stop: the agent boundary, each arm's last telemetry event — 2026-08-25T18:01:42Z (sonnet-control), 2026-08-25T14:49:25Z (sonnet-pushed), 2026-08-23T23:56:22Z (gpt-control), 2026-08-23T23:06:27Z (gpt-pushed). Use it for anything scoped to what the agent itself did. Between the two, the delegated Kamibots signer keeps working with the arm's stored operator key — on sonnet-control it signed 8 further transactions after the agent's last session, up to 19:23:56Z. Those are real chain activity inside the run window, but they are the delegate's, not the agent's. Pick one boundary per question and say which; never mix them in a single figure.

The extracts' lower bound is block_timestamp >= 2026-08-17T20:10:00Z — they also carry the pre-run history of purchased kamis under their previous owners.

Standing reconciliation traps, all reproduced this run:

• —Telemetry ok=true on a submitted transaction does not imply on-chain success; receipts decide.
• —executeBatchedAllowFailure outer transactions record intent, not effect — an outer success can contain a failed inner call.
• —A multi-transaction tool can land one leg and lose another under ok=false (the mempool-saturation class above); the landed hash exists only chain-side.
• —kami_id in the oracle rows is a uint256 entity id, not the human-facing kami index.
• —Three tools return no hash at all; level-up rows carry a NULL target_kami_id.
• —Address matching is case-insensitive (LOWER()); amounts are HUGEINT; de-duplicate on tx_hash.

Layout

<arm>/
├── manifest.yaml          # full run config: model, sampling params,
│                          #   pinned SHAs, cache-aware price table,
│                          #   caps, scaffold_profile, rung flags,
│                          #   wallet ADDRESSES
├── telemetry.jsonl        # append-only scaffold event stream (0.5.0)
├── transcripts/
│   └── session-NNNN.jsonl # full model context per session, as sent
└── oracle/
    ├── kami_action_{operator,owner}_terminal.csv
    ├── raw_tx_{operator,owner}_terminal.csv
    └── README.md          # extract provenance + trap list
analysis/
├── TOOL_SCORECARD.md      # per-tool attempt/error census, all four
└── tool_scorecard.csv     #   arms, over the full 101-tool surface
RUN.md                     # the frozen run card: registration,
                           #   pre-registered expectations, box, pins
MANIFEST.json              # sha256 + size for every file above

Not published here (they stay in the private archival master): runner logs, scaffold state files, the agents' workspace folders, and the investigator-side chain-watch state.

Schemas

Transcripts — one JSONL file per session; one message per line, exactly as sent to the model:

• —{"role": "user", "text": ...}
• —{"role": "assistant", "text": ..., "tool_calls": [{id, name, args}]}
• —{"role": "tool_result", "tool_call_id": ..., "content": ..., "is_error": bool}

Assistant rows may carry provider_state (opaque provider-side continuation state where the SDK returns one). Sonnet 5's thinking text is not in these traces — the provider does not return it by default at these pins, so process observables here are tool-level.

Telemetry — one event per line; schema 0.5.0 (scaffold v0.5.1). Event types: run_start, session_start, llm_request, llm_call, tool_call, workspace_write, session_end, schedule_next.

• —run_start — manifest_hash, model, harness_sha, agent_sha, gdd_sha, the full harness_tools list, and the price_table actually used for accounting.
• —session_start — scaffold_profile (`control` or `pushed` — the experiment's variable, recorded every session), harness_tools_hash, tools_hash, presentation_mode, trigger, budget_remaining_usd, wallclock_elapsed_s.
• —llm_request / llm_call — write-ahead pairing on request_seq (a request with no paired call is a crash marker). The call row carries input_tokens, output_tokens, cache_read_tokens, cache_write_tokens (+ 5m/1h split), cost_usd, cumulative_tokens, cumulative_usd, stop_reason, latency_ms, retry_count, provider_request_id.
• —tool_call — tool, source, initiator (model vs scaffold), call_seq, ok, duration_ms, provider_call_id, plus, where applicable: tx_hash / txs[] and tx_terminal_state for submitting tools; error; lens_stale / lens_block for world-state reads; truncated / original_bytes; path for workspace operations; and `query` / `hits` on `search_reference` — the pushed rung's search path, which is why these fields appear only on pushed arms.
• —workspace_write — path, bytes, workspace_total_bytes.
• —session_end — reason (the stop code), llm_calls, tool_calls, session_tokens, session_cost_usd.
• —schedule_next — requested_min (what the agent asked for), clamped_min (what it got), next_wake_at, source.

Budget fields were never visible to the agents.

Oracle extracts — decoded from chain by the investigator-side oracle service (never accessible to the agents); rows for each arm's operator AND owner wallets. Kami purchases are owner-signed — merge both when reconstructing acquisitions. raw_tx_* are transactions (including reverts); kami_action_* are the decoded game actions. Each arm's oracle/README.md carries that extract's exact window, provenance, and trap list.

Tool scorecard — generated from the committed telemetry alone by the lab's standing script: per tool and per arm, attempts / ok / errors / error rate / longest failure streak / top error class, plus flags (F-ZERO-OK never succeeded, F-HIGH-ERR, F-RETRY-LOOP, F-NEVER never attempted). It is a harness-ergonomics instrument, not an outcome measure.

Provenance and integrity

Everything is pinned in each arm's manifest.yaml: scaffold (`tokedo/kami-agent` v0.5.1, e42fd9a3ac48005a9fa588a7ac22f3285b2ee743), environment interface (`tokedo/kami-harness` 2.2.0, 7c6a7bab79e08dececdcfdcb2cf04e88e73e5958 — 101 tools, registry mass 69,900, surface byte-identical under both rung flag values), world-state reader (`tokedo/kami-lens` 0.4.0, 1d7a96082510e7cbb4e0eb7ac8a4a5fd4deafce3), and the game-documentation snapshot (`tokedo/kamigotchi-gdd`, 6746314981f9a094dc1a71f74d6d5d0d1b8446d9), plus model strings, sampling parameters, caps, and the cache-aware price tables used for budget accounting:

Sampling: Sonnet 5 at the provider default (adaptive thinking, no thinking field sent); GPT-5.2 at reasoning_effort: medium; max_tokens 16,000 per call on both; no temperature sent. The reasoning settings are asymmetric between providers by necessity (each model "thinking-on as shipped") — recorded, not controlled.

Cost recomputation from per-call telemetry matches each arm's terminal state exactly (verified at every monitor pass and at both close-outs). Quest, level-up and purchase counts come from chain state, not from self-report. MANIFEST.json carries sha256 and byte size for every published file. cumulative_tokens counts input + output only.

Known caveats

• —N=1 per cell, one seed, one world epoch, two models — case-study evidence, not statistics. Evidence tier EXPLORATORY.
• —The pushed rung is a bundle (orientation + search + enriched perception + error snippets). Wave 1 was designed to bisect it only if Wave 1 moved. It did not, so the bundle stays unbisected.
• —No rung-0 arm (no arm without the workspace folder).
• —Both pairs were operator-stopped before their walls, at different days (6.1 and 8.0), for the reasons recorded above.
• —Live shared world: other human players acted throughout, and the arms' windows differ slightly. Zero third-party actions touched study entities this run.
• —This is not a model ranking. The pair is matched, not raced; the comparison that carries meaning is each model against its own control.
• —The delegation service's status endpoint is unreliable in both directions — it reported containers RUNNING that never signed, and 404 for containers that then started successfully. Only the receipt census decides.
• —Wallet addresses and in-game account ids appearing throughout are public by design; the cohort-identity embargo held for the duration of the live run and lifts with this release. The wallets were temporary, swept, and retired at experiment close.

License and citation

CC-BY-4.0. Cite the dataset by its pinned revision tag (v0-final); later corrections are ordinary commits that cannot move the tag.