ServiceNow-AI/erpnext-tasks

ERPNext Tasks

An evaluation dataset for web agents operating on a self-hosted ERPNext instance. Each row contains a task goal, a configuration dict with a SQL validation query, and a standalone Python validation function that scores agent performance by querying the ERPNext database.

Dataset overview

• —207 samples across 82 task categories
• —A single shared validator queries a SQL endpoint to verify outcomes

Schema

Validation

The validator module exposes a validate(messages, config) function:

def validate(messages: List[Dict[str, str]], config: Dict[str, Any]) -> int:
    """
    Args:
        messages: Conversation history (list of {"role": ..., "content": ...} dicts).
        config:   The task_config dict for this sample, containing:
                  - sql: A SQL query to run against the ERPNext database
                  - count: The expected results_count value

    Returns:
        1 if the task was completed successfully, 0 otherwise.

    Raises:
        ValueError: If the task configuration is invalid.
    """

The validator calls a SQL query endpoint on the ERPNext instance. The following environment variables must be set:

Usage

import ast
from datasets import load_dataset

ds = load_dataset("servicenow-ai/erpnext-tasks")

sample = ds["train"][0]
config = ast.literal_eval(sample["task_config"])

# Load the validator
exec(sample["validation"], globals())
score = validate(
    messages=[{"role": "assistant", "content": "Done."}],
    config=config,
)

Task categories

User Management

• —assign_role: Assign a role profile to a user
• —create_user_email_first_last: Create a user with email, first name, and last name
• —create_user_email_first_last_username_phone: Create a user with full contact details
• —create_user_with_email_account_dependency_HARD: Create a user with email account dependency
• —create_user_group_with_new_users_HARD: Create a user group with newly created users
• —disable_user_account: Disable a user account
• —create_user_then_cancel_journal_entry: Create a user then cancel a journal entry

Purchasing

• —create_supplier_name_email_phone: Create a supplier with contact details
• —create_supplier_name_group_currency_tax_id: Create a supplier with financial details
• —create_purchase_order_with_discount / without_discount: Create purchase orders
• —create_purchase_order_then_manage_supplier: Create a purchase order then manage the supplier
• —create_stock_item_with_name_description_then_create_purchase_order: Create item then order
• —create_stock_item_without_name_description_then_create_purchase_order: Create item then order (minimal)
• —add_date_confirmation_number_to_purchase_order: Update purchase order confirmation details
• —add_comment_to_draft_purchase_orders: Add comments to draft purchase orders

Sales

• —create_customer_name_account_manager_group: Create a customer with manager and group
• —create_customer_name_type_address: Create a customer with address details
• —create_customer_with_account_manager_group_tax_dependencies_HARD: Complex customer creation
• —create_customer_with_account_manager_role_profile_group_tax_dependencies_HARD: Complex customer creation with role profiles
• —create_sales_order / create_sales_order_put_on_hold: Create and manage sales orders
• —create_sales_invoice / create_sales_invoice_with_custom_tax_rate: Create invoices
• —create_sales_invoice_with_date_cost_center: Create invoice with cost center
• —create_sales_invoice_with_customer_item_account_dependencies_HARD: Complex invoice creation
• —create_sales_invoice_with_customer_item_territory_pricelist_account_dependencies_HARD: Complex invoice with territory and pricelist

Inventory

• —create_stock_item_id_group: Create a stock item with ID and group
• —create_stock_item_name_id_group_description: Create a stock item with full details
• —create_stock_item_name_id_group_description_shelf_life_warranty: Create item with shelf life and warranty
• —create_stock_entry / create_stock_entry_with_letterhead: Create stock entries
• —edit_stock_item_warranty_weight: Edit item warranty and weight
• —edit_stock_item_warranty_weight_material_request_type_tag: Edit item with additional fields
• —delete_stock_item / disable_stock_item: Remove or disable stock items

HR

• —create_employee_name_gender_dob_status_join_date: Create employee with basic info
• —create_employee_name_gender_dob_join_date_salary_phone: Create employee with salary and phone
• —create_employee_full_address_HARD: Create employee with full address details
• —create_employee_with_emergency_contact_HARD: Create employee with emergency contact
• —update_employee_department: Update an employee's department

Accounting

• —create_journal_entry / create_journal_entry_with_account_dependency_HARD: Create journal entries
• —create_payment_entry_without_cost_center / with_cost_center_cheque_date_reference: Create payment entries
• —create_payment_entry_with_customer_account_project_dependencies_HARD: Complex payment entry
• —create_payment_entry_with_mode_of_payment_account_supplier_cost_center_dependencies_HARD: Complex payment entry with supplier
• —cancel_journal_entry / cancel_payment_entry / cancel_purchase_order: Cancel accounting documents
• —assign_journal_entry / assign_payment_entry / assign_purchase_order: Assign documents to users

Projects & Tasks

• —create_project_with_name_company: Create a project with name and company
• —create_project_with_name_company_priority_note: Create project with priority and notes
• —create_project_with_name_company_priority_note_dept_cost: Create project with department and cost center
• —create_project_with_customer_full_address_HARD: Create project with customer address
• —create_project_with_type_dept_dependencies_HARD: Create project with type and department dependencies
• —create_project_with_type_dept_cost_center_dependencies_HARD: Create project with cost center dependencies
• —create_project_with_type_dept_customer_cost_center_dependencies_HARD: Create project with full dependencies
• —create_task_with_subject_project: Create a task in a project
• —create_task_with_subject_project_priority_dept: Create task with priority and department
• —create_task_with_subject_project_priority_dept_comment: Create task with comment
• —create_task_with_project_type_dept_dependencies_HARD: Create task with dependencies
• —create_task_with_project_type_parent_dept_dependencies_HARD: Create task with parent task dependency
• —change_project_status / delete_project / delete_task: Manage project lifecycle
• —update_task_status / assign_task: Update and assign tasks

Multi-step & Cross-domain

• —assign_sales_invoice / tag_sales_invoice / favorite_sales_invoice: Manage sales invoice metadata
• —tag_supplier / delete_supplier: Manage supplier metadata
• —update_target_warehouse_on_suppliers_purchase_orders: Update warehouse on purchase orders
• —create_customer_then_create_sales_invoice: Create customer then invoice
• —create_customer_then_create_sales_order: Create customer then sales order
• —create_sales_invoice_then_assign_task: Create invoice then assign a task
• —create_project_then_cancel_payment_entry: Create project then cancel payment entry
• —create_task_then_assign_payment_entry: Create task then assign payment entry
• —assign_journal_entry_then_update_task_status: Assign journal entry then update task