CoolFace
Datasetpublic

ServiceNow/AgentHorizon

AgentHorizon AgentHorizon is a benchmark for evaluating LLM judges of computer-use agents. Each item is a recorded trajectory of a GUI agent attempting a long-horizon desktop task (often 100 to 300+ steps), paired with a ground-truth label that says whether the agent actually completed the task. A judge reads the trajectory (instruction, action sequence, and screenshots) and predicts success or failure. Repository contents Path What it is… See the full description on the dataset page: https://huggingface.co/datasets/ServiceNow/AgentHorizon.

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes1.2kdownloads
Dataset Card

AgentHorizon

AgentHorizon is a benchmark for evaluating LLM judges of computer-use agents. Each item is a recorded trajectory of a GUI agent attempting a long-horizon desktop task (often 100 to 300+ steps), paired with a ground-truth label that says whether the agent actually completed the task. A judge reads the trajectory (instruction, action sequence, and screenshots) and predicts success or failure.

Repository contents

PathWhat it is
AgentHorizon.jsonlLabels for the main split (605 items).
AgentHorizon-Simple.jsonlLabels for a simpler split (768 items).
sandbox/The trajectories themselves (markdown, structured JSON, screenshots) plus the judge prompt and framework.

The two splits differ in difficulty. AgentHorizon-Simple.jsonl holds items that a strong open-weight agent already solves consistently. AgentHorizon.jsonl holds the harder items, where judges disagree and capability headroom remains. The two files do not overlap.

Label files

Each line in a *.jsonl file is one trajectory's ground truth. Labels are keyed by trajectory_id, which is also the filename of the trajectory inside the sandbox.

Every row has the same seven fields. A negative shows one deliverable's instruction over another deliverable's trajectory. A positive shows one deliverable's own instruction over its own trajectory, so instruction_id and steps_id are the same and the *_source and mistake_type fields are empty.

Positive example:

json
{"trajectory_id": "010f92ae-b7a4-48ba-aca7-5d3642754295", "label": "positive", "instruction_id": "000234004-234004-0000-000000234004", "instruction_source": "", "steps_id": "000234004-234004-0000-000000234004", "steps_source": "", "mistake_type": ""}

Negative example:

json
{"trajectory_id": "00159373-ed47-4798-8ff5-92c179e00179", "label": "negative", "instruction_id": "000233081-233081-0000-000000233081", "instruction_source": "child", "steps_id": "000233374-233374-0000-000000233374", "steps_source": "parent", "mistake_type": "Bad Side Effect"}
FieldDescription
trajectory_idUUID. Item identity, and the filename of the trajectory at sandbox/data/markdowns/<trajectory_id>.md and sandbox/data/jsons/<trajectory_id>.json.
label"positive" (agent completed the task) or "negative" (it did not).
instruction_idDeliverable ID whose instruction is shown. For positives, the same as steps_id.
instruction_sourcePair-role of that instruction: "parent" or "child". Empty for positives.
steps_idDeliverable ID whose trajectory (steps and screenshots) is shown. Its screenshot folder is sandbox/data/media/images/<steps_id>/. For positives, the same as instruction_id.
steps_sourcePair-role of that trajectory: "parent" or "child", always the opposite of instruction_source. Empty for positives.
mistake_typeFor negatives, the kind of failure: "Critical Mistake", "Bad Side Effect", or "Misunderstanding of the Instructions". Empty for positives, and for a few negatives where the category was not recorded.

Counts:

FilePositiveNegativeTotal
AgentHorizon.jsonl287318605
AgentHorizon-Simple.jsonl236532768

Paired construction (how negatives are built)

Tasks come in sibling pairs: a parent task and a child variant whose instruction differs from the parent in one detail. The two share the same applications and a near-identical step count. A negative is built by showing one sibling's recorded trajectory together with the other sibling's instruction. The trajectory is a coherent, successful execution, just of the wrong instruction, so a judge that only checks surface coherence is fooled. steps_id identifies the deliverable whose trajectory and screenshots are shown, with steps_source its pair-role; instruction_id identifies the sibling whose instruction was substituted, with instruction_source its pair-role (always the opposite); mistake_type records how the mismatch manifests.

This is what makes the negative half adversarial. The failure lives in the alignment between instruction and trajectory, not in any visible error in the trajectory itself.

Mistake types

  • —Critical Mistake: the core objective was not achieved. Irreversible; the user would have to restart.
  • —Bad Side Effect: the goal was achieved but with unwanted consequences the user must undo (cost, risk, or significant cleanup).
  • —Misunderstanding of the Instructions: the agent took a reasonable action but misread a detail. A small, contained fix.

Sandbox

sandbox/ is a self-contained working directory for running a judge over the trajectories.

sandbox/
  AGENTS.md / CLAUDE.md                    judge framework (identical copies for different harnesses)
  prompt.md                                per-trajectory judge prompt (templated on {{TRAJECTORY_ID}})
  data/
    markdowns/<trajectory_id>.md           trajectory as markdown (goal + numbered actions + inline screenshots)
    jsons/<trajectory_id>.json             the same trajectory in structured form (task, environment, steps[])
    media/images/<steps_id>/step_N.png  screenshots referenced by the markdown and JSON

To judge one trajectory, set the working directory to sandbox/, read data/markdowns/<trajectory_id>.md, and follow prompt.md and AGENTS.md. Screenshot links inside each markdown and JSON are written relative to that file, for example ../media/images/<steps_id>/step_N.png, so they resolve correctly when the file is opened from data/markdowns/ or data/jsons/.

The ground-truth label for a trajectory_id lives in the JSONL files at the repository root. It must not be shown to the judge during evaluation.

ServiceNow/AgentHorizon · CoolFace