SamuelChien821/devopsbench-100

DevOpsBench-100

DevOpsBench-100 is a synthetic long-horizon software-engineering / SRE agent benchmark: 100 tasks over one executable world ("NovaCart", a mid-size e-commerce SaaS) with 72 SQLite tables, 1451 seeded rows, a 38-file monorepo with 417 commits, and 97 MCP tools spanning a first-party engineering stack (tickets, PRs, CI, deployments, canaries, migrations, feature flags, metrics, alerts, incidents, chat, knowledge base) plus deliberately disagreeing vendor-shaped surfaces (Jira, Linear, GitHub Issues, Prometheus, Sentry, PagerDuty, Confluence, spreadsheets) and Kubernetes.

Tasks are high-level workplace requests: the employee states the operational question or desired business result while procedure, authority, and causal facts remain distributed across the sandbox. Reference trajectories run 32-67 tool calls (median 35), with 100/100 distinct tool-name sequences. Every task publishes 53 task-scoped native assets: 32 contextual workplace artifacts, 18 exact non-empty sandbox exports that correspond one-to-one with the decision-controlling reads, and one inspectable asset manifest. The reference performs 26-34 context reads, while the verifier requires all 20 decision-controlling joins, the task-specific state transitions, a derived capacity answer with three date/cost/authority options, provider readback, and a reopened handoff. Acceptance is fully deterministic and expressed as 16 task-specific semantic milestones totaling 100 points. Low-level vcode, final-state, sequence, and anti-forgery checks remain nested verifier evidence - no LLM judge, network, or clock is in the reward path.

What is included

• —data/tasks.jsonl: task records (task_id, task_name, world_id, prompt, context_files, rubric, gold_output, metadata).
• —tasks/: one readable JSON record per task (includes the guided instruction variant).
• —task_files/: 53 inspectable assets per task spanning tickets, services, observability, incidents, deployments, code, CI, migrations, vendor trackers, knowledge, chat, approvals, capacity reservations, vendor lead times, and customer commitments. The material/ subdirectory contains the 20 exact sandbox exports used by the deterministic causal contract; the manifest records source, query scope, material reason, format, byte size, and digest for every listed file without disclosing an execution order or gold state.
• —world/: the offline world source - stdlib MCP server, tool implementations, seeded SQLite database, schema and seed SQL.
• —verifiers/: 100 standalone verifier scripts (python3 verify_<task>.py world.db prints the full verdict).
• —trajectories/: one executed reference trajectory per task (JSONL).
• —reports/: measured build and qualification evidence.

Task families (19)

Objective release gates

Provenance and contamination

Every service, metric, document, commit, and incident is synthetic and was generated for the NovaCart world. 30 cross_system/handover tasks were ported from TheAgentCompany task shapes re-grounded onto NovaCart's own state; the AIOps families reproduce the microsoft/AIOpsLab task structure (detect / localize / analyze) against NovaCart. No third-party benchmark text, evidence, or gold answers are included. Gold outputs are public, so this release is appropriate for transparent evaluation and RL experiments rather than secret-test claims.

Licenses

Synthetic task data and world content are CC-BY-4.0. Benchmark code, server, and verifiers are Apache-2.0.