ServiceNow-AI/EnterpriseOps-Gym

<div align="center">

<h1><img src="assets/csmgym.png" alt="Logo" width="48" style="vertical-align:middle; margin-right:8px;" /> EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings</h1>

<p> <a href="https://enterpriseops-gym.github.io/"><img src="https://img.shields.io/badge/Website-blue?logo=google-chrome&logoColor=white" /></a> <a href="https://arxiv.org/abs/2603.13594"><img src="https://img.shields.io/badge/Paper-red?logo=arxiv&logoColor=white" /></a> <a href="https://github.com/ServiceNow/EnterpriseOps-Gym"><img src="https://img.shields.io/badge/GitHub-black?logo=github" /></a> </p>

<p><i>EnterpriseOps-Gym is a containerized, resettable enterprise simulation benchmark for evaluating LLM agents on stateful, multi-step planning and tool use across realistic enterprise workflows</i></p>

</div>

<div align="center"><img src="assets/teaser.png" alt="EnterpriseOps-Gym Overview" width="80%" /></div>

About

EnterpriseOps-Gym is a large-scale benchmark for evaluating the agentic planning and tool-use capabilities of LLM agents across enterprise operations. It comprises 1,150 expert-curated tasks spanning 8 enterprise domains, each running against live containerized MCP servers backed by realistic, fully synthetic databases.

Unlike static QA benchmarks, EnterpriseOps-Gym evaluates agents on final environment state using SQL verifiers - meaning agents are rewarded for achieving the correct outcome, not for following a rigid action sequence. Tasks require long-horizon multi-step reasoning, strict policy compliance, and precise tool invocation under complex data dependencies.

Best model performance: 34.1% success rate - leaving significant headroom for future research.

Key Features

• —🛠️ 512 tools across 8 enterprise domains
• —🗄️ 164 database tables with avg 1.7 foreign-key dependencies per table
• —🔢 9.15 avg steps per task (up to 34), with 5.3 avg verification conditions
• —📏 89k avg context length per task
• —🔒 Tasks enforce access control, policy compliance, and referential integrity
• —✅ Evaluation is outcome-based via executable SQL verifiers — not action-sequence matching
• —🐳 Fully containerized sandbox — reproducible and isolated per task run

Evaluation Framework

The evaluation code is available at ServiceNow/EnterpriseOps-Gym.

The framework supports:

• —Multiple orchestrators: ReAct, Planner-ReAct, Decomposing Planner
• —Multiple LLM providers: Anthropic, OpenAI, Azure OpenAI, Google Gemini, DeepSeek, vLLM, and more
• —Parallel execution via Ray for large-scale runs
• —Automatic scoring with per-task and per-mode breakdowns

from datasets import load_dataset

ds = load_dataset("ServiceNow-AI/EnterpriseOps-Gym", "oracle", split="teams")

Domain Information

The dataset is organized by domain (split) and mode (configuration subset).

Domains

Modes (Tool-Set Configurations)

Each mode controls the set of tools exposed to the agent, simulating realistic tool-retrieval scenarios:

Field Descriptions

Each row in the dataset corresponds to one task instance and contains the following fields:

Example Use Cases

EnterpriseOps-Gym can be used for:

• —Benchmarking LLM agents on realistic enterprise workflows across IT, HR, CRM, and collaboration domains
• —Evaluating tool-use and planning under long-horizon, multi-step, policy-constrained settings
• —Studying tool retrieval robustness by comparing oracle vs. distractor-augmented tool modes
• —Developing new orchestration strategies — the framework natively supports ReAct, Planner-ReAct, and Decomposing Planner
• —Studying failure modes of state-of-the-art models on high-complexity enterprise tasks (best model: 34.1%)
• —Extending the benchmark with new domains, tasks, or verifiers using the released Docker sandbox infrastructure

Citation

@misc{malay2026enterpriseopsgymenvironmentsevaluationsstateful,
      title={EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings}, 
      author={Shiva Krishna Reddy Malay and Shravan Nayak and Jishnu Sethumadhavan Nair and Sagar Davasam and Aman Tiwari and Sathwik Tejaswi Madhusudhan and Sridhar Krishna Nemala and Srinivas Sunkara and Sai Rajeswar},
      year={2026},
      eprint={2603.13594},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2603.13594}, 
}