jayoohwang/ebrowse-v1.1-online-mind2web-subset

ebrowse v1.1 — Online-Mind2Web subset

20 full browsing-agent traces produced by `ebrowse`, a token-efficient browser-control CLI for LLM agents. Each trace records a complete task attempt on a live public website: every tool call, every tool output, a screenshot and DOM snapshot per step, browser/console/network events, and the agent's own messages.

These are raw research traces, not a cleaned benchmark. They exist so the ebrowse trace viewer can replay exactly what the agent saw and did.

Viewing the traces

The traces are meant to be read in the ebrowse trace viewer, which renders each run as a conversation with an aligned lane of post-action browser state.

# 1. clone and set up ebrowse
git clone https://github.com/jayoohwang1/ebrowse.git
cd ebrowse
make setup

# 2. download this dataset
hf download jayoohwang/ebrowse-v1.1-online-mind2web-subset \
  --repo-type dataset --local-dir /tmp/ebrowse-traces

# 3. drop the run group into the repo's runs/ folder
mkdir -p runs
cp -r /tmp/ebrowse-traces/online-mind2web-subset runs/

# 4. browse them
uv run ebrowse-eval serve runs --open

The viewer discovers runs by searching for events.jsonl, and names each group after its parent directory — so the runs appear under an `online-mind2web-subset` group. You can rename that directory to rename the group; nothing inside depends on its name.

Inspecting a single run from the CLI

R=runs/online-mind2web-subset/b4aa7315e31dfcdc52baf7771be260c9

uv run ebrowse-eval overview  $R   # task, outcome, per-step table
uv run ebrowse-eval errors    $R   # steps that failed, and whether recovery hints were followed
uv run ebrowse-eval issues    $R   # LLM annotations: verdict, issue spans, vision findings
uv run ebrowse-eval step      $R 23  # everything recorded for one step
uv run ebrowse-eval trace-ref $R @e307  # the full history of one element ref
uv run ebrowse-eval view      $R --open  # single-run HTML page

What is in each run directory

<task-id>/
├── events.jsonl   # the trace: run_meta, agent_message, step, browser_event,
│                  # ebrowse_log, anomaly, summary, run_end records
└── blobs/         # content-addressed payloads: <hex[:2]>/<hex>.{png,json}
                   # referenced from events.jsonl as "sha256:<hex>"

events.jsonl is newline-delimited JSON, one record per line, each with a type field. The schema is documented in the ebrowse repo at evals/docs/trace-schema.md, and ebrowse-eval validate <run-dir> checks a run against it.

Loading it directly, without ebrowse:

import json
records = [json.loads(l) for l in open("events.jsonl")]
steps = [r for r in records if r["type"] == "step"]
print(steps[0]["command"], "->", steps[0]["output"][:200])

How these were produced

The model is a modest local one, chosen deliberately: it makes tool-UX problems visible. Several traces show the agent defeated by the interface rather than by the task.

Outcomes

outcome and eval.success in each run's run_end record were assigned by manual review, because these tasks ship no automatic evaluator. Every verdict records its provenance in eval.details.source:

• —manual-review:trace — verified from trace evidence (final URL carries the filters, site confirmation banner, calendar clicks corroborated by screenshots, …)
• —manual-review:user — adjudicated by a human where the trace alone could not settle it

Note that outcome keeps timeout for runs that hit the wall clock, matching the harness's own precedence (a terminal condition outranks the eval result); those runs carry eval.success = false.

10 succeeded / 10 failed (6 of the failures by timeout).

Annotations

19 of the 20 runs carry post-hoc annotations from a local model (ebrowse-eval annotate), stored as summary records with kind ∈ {verdict, issue, stuck_span, vision} and rendered by ebrowse-eval issues. 949dc965… has none: the annotator failed to emit a parseable verdict for that run.

Treat the annotations as untrusted. They were produced by the same small model that drove the runs, and they are demonstrably unreliable about outcomes — several verdicts claim the agent "failed to report the final answer" when the transcript contains a complete answer. They appear to over-index on the unknown outcome that these runs originally carried. They are useful as pointers to interesting step ranges, not as ground truth. The run_end eval verdicts above were assigned by human review and supersede them.

Known content of the traces

• —All sites are public and were browsed logged-out; no accounts were used.
• —Task inputs are synthetic (fictional dates, a fictional child, generic cities).
• —Local filesystem paths have been rewritten to /ebrowse and /home/user.
• —Browser caches, spooled captures, and raw harness logs were removed; what remains is the trace and the blobs it references.

Provenance of the task set

Tasks come from the Online-Mind2Web dataset (hard subset). Task prompts belong to that project; the traces here are new.