ServiceNow-AI/servicenow-tasks

ServiceNow Tasks

An evaluation dataset for web agents operating on ServiceNow instances. Each row contains a task goal, a configuration dict, and a standalone Python validation function that scores agent performance by querying the ServiceNow API.

Derived from the WorkArena L1 benchmark.

Dataset overview

• —330 samples across 33 task types grouped into 8 categories
• —Each task has 10 seeded variants
• —Validators are self-contained Python functions that call the ServiceNow REST API to verify outcomes

Schema

Validation

Each validator module exposes a validate(messages, config) function:

def validate(messages: List[Dict[str, str]], config: Dict[str, Any]) -> int:
    """
    Args:
        messages: Conversation history (list of {"role": ..., "content": ...} dicts).
                  The last assistant message is used for answer-based tasks.
        config:   The task_config dict for this sample.

    Returns:
        1 if the task was completed successfully, 0 otherwise.

    Raises:
        ValueError: If the task configuration is invalid (not the agent's fault).
    """

Validators for create, filter, sort, order, and dashboard tasks call the ServiceNow REST API to verify outcomes. The following environment variables must be set:

Usage

import ast
from datasets import load_dataset

ds = load_dataset("servicenow-ai/servicenow-tasks")

sample = ds["train"][0]
config = ast.literal_eval(sample["task_config"])

# Load the validator
exec(sample["validation"], globals())
score = validate(
    messages=[{"role": "assistant", "content": "The answer is 42."}],
    config=config,
)

Task categories

All menu

Navigate to a target module (e.g. Performance Analytics > Indicator Groups). Validated by checking the final URL against the expected path.

Create

Create a new record on a specific table with prescribed field values. Validated by querying the ServiceNow Table API to check that the record exists with the correct field values.

Filter

Apply AND/OR filters on a list view. Validated by querying the ServiceNow Table API with the expected filter conditions and comparing the result set with the currently displayed list.

Sort

Sort a list view by one or more columns in ascending or descending order. Validated by querying the ServiceNow Table API with the expected sort order and comparing against the displayed list.

Order

Order an item from the service catalog with a specific quantity and configuration. Validated by querying the ServiceNow Table API to check that the catalog request was submitted with the correct item, quantity, and configuration options.

Impersonation

Impersonate a specific user. Validated by checking the ServiceNow session API for the impersonated user.

Knowledge base search

Answer a question using ServiceNow's knowledge base. Validated by checking that the agent's answer contains the expected string (or one of the alternative answers).

Dashboard / Chart retrieval

Read a value, percentage, min, max, mean, median, or mode from a chart on a dashboard or report. Validated by fetching chart data from the ServiceNow Stats API and comparing against the agent's answer. Supports single-series charts, multi-series charts with chart_series filtering, and trend charts with time-period bucketing.