Datasetpublic

harry1332/agentbench-avalon-sampled-cpu-trace

AgentBench Avalon: sampled CPU trace One completed non-coding AgentBench episode, with one validated CPU trace sample. This release contains both raw and converted DynamoRIO traces, readable game/model logs, exact collection code, and checksums. It was collected on September 23, 2026. Execution completed normally; the agent lost the game. These are different outcomes. This single run is not a benchmark win-rate estimate. Start here Reading guide: what to open… See the full description on the dataset page: https://huggingface.co/datasets/harry1332/agentbench-avalon-sampled-cpu-trace.

sourceHugging Faceupdated 2d agoView on Hugging Face

0likes44downloads

Dataset Card

AgentBench Avalon: sampled CPU trace

One completed non-coding AgentBench episode, with one validated CPU trace sample. This release contains both raw and converted DynamoRIO traces, readable game/model logs, exact collection code, and checksums. It was collected on September 23, 2026.

Execution completed normally; the agent lost the game. These are different outcomes. This single run is not a benchmark win-rate estimate.

Start here

—Reading guide: what to open, download, and run.
—Readable game log and model responses.
—Trace validation log and sample metadata.
—Agent protocol changes: how the earlier parser failure was fixed.

What was run?

Item	Result
Benchmark	AgentBench `avalon-dev-single`, development episode 0, seed 7
Task type	Non-coding social-deduction game; one LLM player, four built-in baseline players
Official status	`completed`; process exit 0
Agent model	Qwen3-Coder-30B-A3B-Instruct-AWQ, separate GPU server
Model calls	26; zero rejected responses and zero retries
Completion tokens	5,824
Episode duration	247.491 seconds, excluding Python startup/imports
Game outcome	LLM Player 0 (Servant) lost; Evil won by mission
Deduction accuracy	0.4
CPU sample	9,994,930 fetched instructions; invariant checks passed

Sampling methodology

Skip the first 500M instructions. Trace 10M, then execute 990M without detailed tracing, and repeat. Intended starts are 500M, 1.5B, 2.5B, etc. The collector uses native -trace_after_instrs, -trace_for_instrs, -retrace_every_instrs, and -split_windows. No full execution trace or Scarab modification is involved. This episode emitted only the first window; a separate smoke test verified repetition, but those test traces are not part of this dataset.

The nominal sample range is [500,000,000, 510,000,000). DynamoRIO's thresholds are approximate; the measured count is 9,994,930. Exact global endpoints were not measured. The first 500M are skipped application execution, not recorded simulator warmup. Python startup/imports count toward this skip.

Scope is the CPU process containing the agent adapter and Avalon environment. GPU model inference, a distributed instruction timeline, and Scarab simulation results are not included. Approximately 1% steady-state instruction coverage does not establish a measured runtime-overhead percentage.

Files and sizes

text

README.md                         Overview and outcomes
docs/                             Reading guide and protocol changes
results/                          Official result, readable logs, full model-call JSONL
traces/sample_0000_instr_500000000_510000000/
  metadata.json                   Nominal range and measured count
  drmemtrace.python3.11.00361.2764.dir/
    raw/                          Raw window plus shared module/encoding metadata
    trace/window.0000/            Converted binary trace and scheduling records
validation/                       Actual counts, invariant log, export verification
code/                             Exact collector, agent adapter, tests, run command
provenance/                       Revisions, dependencies, source-file mapping
SHA256SUMS                        Checksums of every release file except this list

Trace file	Size (decimal MB)
Converted `.trace.zip`	27.15
Raw `.raw.lz4`	9.57
Converted binary records if uncompressed	178.74

Converted means tool-readable binary, not a plain-text instruction listing. The README, game log, model responses, JSON metadata and validation log are readable directly in the browser. The reading guide explains how to render CPU records as text. Files are provided directly, without requiring nested archive extraction.

Download and verify

bash

hf download harry1332/agentbench-avalon-sampled-cpu-trace --repo-type dataset --local-dir avalon-trace
cd avalon-trace
sha256sum -c SHA256SUMS
# On macOS: shasum -a 256 -c SHA256SUMS

Use DynamoRIO 11.91.20708 on Linux x86-64 for reading/validation. Start with the converted trace; reconverting raw data may need original application binaries. Preserve the process directory layout. Original execution paths in provenance records refer to the Phoebe container, not your download directory.

Reproducibility and limitations

AgentBench source is pinned to ed013ff9887b0c3d7864c56ae54d41eba54a99d8 (v0.2) and was checked unmodified. Model snapshot: QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ at c58857a7f41c0920f73d1b56678640f9c02017d7. Generation: temperature 0, seed 7, 768-token cap, repetition penalty 1.1. The agent adds format-only instructions for structured extraction, validates outputs, and permits at most three attempts; no retries were needed in this completed run. Game rules and scoring are unchanged.

The first run failed on trailing prose in a probability answer; a second failed on repetitive generation. This is the third attempt, not an unreported first-try success. Earlier failures are described in the protocol notes; the second-attempt result is included under provenance. Full earlier traces are not duplicated here.

All 35 files in the original exported evidence bundle were SHA256-verified against Phoebe. FILE_MAPPING.json maps published source files to that evidence; SHA256SUMS covers this reorganized release, including newly written documentation.

Source projects: AgentBench, DynamoRIO, Scarab-infra, model. Referenced third-party software and model assets retain their upstream terms. Model weights are not included.