Datasetpublic

renderfy/runopsy-bench

Runopsy-Bench Twenty labelled agent traces for measuring failure-onset localization: given a run that went wrong, which step did it start going wrong at — not which step it stopped at. Produced for Runopsy, an open-source causal failure analysis engine for agent runs. pip install runopsy. Read this first: these traces are synthetic Every case here was generated, not recorded. They are single-fault traces written to exercise a specific failure mode, with the onset… See the full description on the dataset page: https://huggingface.co/datasets/renderfy/runopsy-bench.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes13downloads

Dataset Card

Runopsy-Bench

Twenty labelled agent traces for measuring failure-onset localization: given a run that went wrong, which step did it start going wrong at — not which step it stopped at.

Produced for Runopsy, an open-source causal failure analysis engine for agent runs. pip install runopsy.

Read this first: these traces are synthetic

Every case here was generated, not recorded. They are single-fault traces written to exercise a specific failure mode, with the onset declared by construction rather than judged by a person.

That makes them useful for one thing and useless for another:

—Useful for regression testing and for comparing localization strategies on a fixed, reproducible set. The same code always produces the same numbers.
—Useless as evidence that a tool saves anyone time on real work. A benchmark whose author also wrote the engine measures agreement with its own assumptions.

If you want human-labelled traces of real agents, use TRAIL or Who&When instead. Runopsy scores 0.0% on both, and that result is published alongside its 94.4% here — see Results below.

Format

One JSON object per line in runopsy_bench.jsonl:

field	meaning
`name`	case identifier
`category`	failure taxonomy class (`tool_execution`, `state`, `handoff`, …)
`description`	what the case is meant to represent
`onset_step`	the ground truth: sequence number where it started going wrong, or `null` for the healthy case
`affected_steps`	steps the onset plausibly reached
`is_healthy`	true for the one case with nothing wrong, used to measure false positives
`deterministically_detectable`	whether structural analysis alone can reach it
`events`	the trace, as OpenInference-compatible typed events

Events carry run_start, tool_call, llm_call, state_snapshot, memory_op, claim, handoff and run_end kinds with hashes rather than payload text.

Results

Measured with runopsy bench --compare, reproducible offline:

strategy	top-1	top-3	mean step distance
no diagnosis	0.0%	0.0%	—
blame the last failing step (what reading a log achieves)	22.2%	44.4%	3.50
blame the earliest failing step	50.0%	50.0%	1.31
Runopsy deterministic engine	94.4%	100.0%	0.11

False positive rate on the healthy case: 0.0%.

Where the same engine fails, on labelled traces somebody else annotated:

benchmark	onset top-1
TRAIL (expert-labelled SWE-Bench agent traces)	0.0%
Who&When (expert-labelled multi-agent traces)	0.0%

On TRAIL, not one of the 30 annotated onsets carries an error status of any kind — they are formatting mistakes, instruction non-compliance, a wrong assumption about a file path. Runopsy's deterministic layers read exit codes and tool statuses, so they are blind to those by construction. The 94.4% above is on traces where the onset was itself a failure. Both numbers belong on the same page.

Loading

python

import json

cases = [json.loads(line) for line in open("runopsy_bench.jsonl", encoding="utf-8")]
print(cases[0]["onset_step"], len(cases[0]["events"]))

Or with the tool that produced it:

bash

pip install runopsy
runopsy bench --compare        # reproduces the table above

Licence and citation

Apache-2.0.

bibtex

@software{feryad_runopsy,
  author  = {Feryad, Vahit},
  title   = {Runopsy: causal failure analysis for AI agent runs},
  url     = {https://github.com/vahit19/runopsy},
  orcid   = {0000-0002-3282-339X}
}