Datasetpublic

MonikaDvorackova/agent-reliability-traces

Agent Reliability Traces A small synthetic dataset of observable AI agent execution traces annotated with reliability and failure-mode signals. The dataset accompanies the Agent Reliability Lab Hugging Face Space. Dataset purpose The dataset is designed for: prototyping agent-trace evaluation testing deterministic reliability heuristics experimenting with failure-mode classification evaluating tool-use trajectories educational and portfolio use It is not… See the full description on the dataset page: https://huggingface.co/datasets/MonikaDvorackova/agent-reliability-traces.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes22downloads

Dataset Card

Agent Reliability Traces

A small synthetic dataset of observable AI agent execution traces annotated with reliability and failure-mode signals.

The dataset accompanies the Agent Reliability Lab Hugging Face Space.

Dataset purpose

The dataset is designed for:

—prototyping agent-trace evaluation
—testing deterministic reliability heuristics
—experimenting with failure-mode classification
—evaluating tool-use trajectories
—educational and portfolio use

It is not intended as a production benchmark.

Structure

Each record contains:

Field	Description
id	Unique trace identifier
goal	Task assigned to the agent
trace	Observable execution trace
risk_label	Coarse reliability-risk label
failure_mode	Primary detected failure mode
has_verification	Whether explicit verification is present
hastoolfailure	Whether tool execution failures are present
has_loop	Whether repeated tool behaviour is present
hasfinalanswer	Whether an explicit final answer is present
contains_overconfidence	Whether overconfident language is present
notes	Annotation explanation

Risk labels

Current labels include:

—low
—moderate
—elevated
—high

Failure modes

Examples include:

—none
—tool_failure
—loopandtool_failure
—missing_verification
—unsupported_answer
—missingfinalanswer
—overconfidence

Data creation

All examples are synthetic and manually constructed for this project.

They are not production traces and do not contain hidden chain-of-thought.

The dataset includes only observable execution-style events such as plans, tool calls, observations, errors and final answers.

Intended use

This dataset can be used to explore:

—agent reliability analysis
—failure-mode detection
—trace classification
—tool-use evaluation
—verification behaviour
—loop detection
—structured agent monitoring

Relationship to Agent Reliability Lab

The associated Agent Reliability Lab uses deterministic heuristics to analyze observable execution traces.

This dataset provides controlled examples that can be used to test and extend that approach.

Limitations

The dataset is deliberately small and synthetic.

The annotations should not be treated as authoritative measures of agent safety or correctness.

The current traces use a simplified human-readable format and do not represent the event schemas of specific agent frameworks.

Future work

Potential extensions include:

—larger trace collections
—structured JSON trajectories
—LangGraph traces
—tool-call graphs
—multi-agent interactions
—human annotations
—learned failure-mode classifiers
—independently collected evaluation traces

License

Apache-2.0